2016-05-17 22:18:24,751 INFO platform.PlatformRunner (PlatformRunner.java:getPlatformClassesFromClasspath(197)) - found 1 test platforms from classpath
2016-05-17 22:18:24,783 INFO platform.PlatformRunner (PlatformRunner.java:addPlatform(246)) - adding test: cascading.BufferPipesPlatformTest, with platform: hadoop2-tez
2016-05-17 22:18:24,817 INFO tez.Hadoop2TezPlatform (Hadoop2TezPlatform.java:setUp(136)) - not using cluster
2016-05-17 22:18:25,205 WARN util.NativeCodeLoader (NativeCodeLoader.java:<clinit>(62)) - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
2016-05-17 22:18:25,560 INFO platform.TestPlatform (TestPlatform.java:getGlobalProperties(75)) - platform property overrides:
2016-05-17 22:18:25,803 INFO util.Util (Util.java:findMainClass(1374)) - using default application jar, may cause class not found exceptions on the cluster
2016-05-17 22:18:25,804 INFO planner.Hadoop2TezPlanner (Hadoop2TezPlanner.java:initialize(151)) - using application jar: /opt/concurrent/home/teamcity/.gradle/caches/modules-2/files-2.1/cascading/cascading-hadoop2-tez/3.1.0/226af02e24d5195c60e4ebe911d0c7c9780f59f0/cascading-hadoop2-tez-3.1.0.jar
2016-05-17 22:18:25,922 INFO property.AppProps (AppProps.java:getAppID(177)) - using app.id: 3A00513914EB4F21AEEB621C8193D9F2
2016-05-17 22:18:26,110 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [test] executed rule registry: NoHashJoinHadoop2TezRuleRegistry, completed as: SUCCESS, in: 00:00.077
2016-05-17 22:18:26,131 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [test] executed rule registry: HashJoinHadoop2TezRuleRegistry, completed as: SUCCESS, in: 00:00.099
2016-05-17 22:18:26,133 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [test] rule registry: NoHashJoinHadoop2TezRuleRegistry, supports assembly with steps: 1, nodes: 2
2016-05-17 22:18:26,133 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [test] rule registry: HashJoinHadoop2TezRuleRegistry, supports assembly with steps: 1, nodes: 2
2016-05-17 22:18:26,135 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [test] rule registry: NoHashJoinHadoop2TezRuleRegistry, result was selected using: 'default comparator: selects plan with fewest steps and fewest nodes'
2016-05-17 22:18:26,282 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [test] tap forcing step to tez local mode: /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.5.x-tez-0.6.x/build/classes/test/data/lhs.txt
2016-05-17 22:18:26,448 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [test] tap forcing step to tez local mode: /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.5.x-tez-0.6.x/build/test/output/hadoop2-tez/bufferpipesplatform/testIdentityBuffer/identity
2016-05-17 22:18:26,659 INFO util.Version (Version.java:printBanner(82)) - Concurrent, Inc - Cascading 3.1.0
2016-05-17 22:18:26,662 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [test] starting
2016-05-17 22:18:26,662 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [test] source: Hfs["TextLine[['offset', 'line' | long, String]->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.5.x-tez-0.6.x/build/classes/test/data/lhs.txt"]
2016-05-17 22:18:26,663 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [test] sink: Hfs["TextLine[['line']->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.5.x-tez-0.6.x/build/test/output/hadoop2-tez/bufferpipesplatform/testIdentityBuffer/identity"]
2016-05-17 22:18:26,663 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [test] parallel execution of steps is enabled: false
2016-05-17 22:18:26,663 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [test] executing total steps: 1
2016-05-17 22:18:26,663 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [test] allocating management threads: 1
2016-05-17 22:18:26,664 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [test] starting step: (1/1) ...stIdentityBuffer/identity
2016-05-17 22:18:26,664 WARN flow.Flow (BaseFlow.java:logWarn(1516)) - [test] 'yarn.timeline-service.enabled' is disabled, please enable to capture detailed metrics of completed flows, this may require starting the YARN timeline server daemon
2016-05-17 22:18:26,689 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [test] tez session mode enabled: true
2016-05-17 22:18:26,861 INFO client.TezClient (TezClient.java:<init>(153)) - Tez Client Version: [ component=tez-api, version=0.6.2, revision=c04283bf91fd96823eec37056d5c8b1a92728265, SCM-URL=scm:git:https://git-wip-us.apache.org/repos/asf/tez.git, buildTime=20150804-1436 ]
2016-05-17 22:18:26,870 INFO client.TezClient (TezClient.java:start(333)) - Session mode. Starting session.
2016-05-17 22:18:26,870 INFO client.TezClientUtils (TezClientUtils.java:setupTezJarsLocalResources(159)) - Ignoring 'tez.lib.uris' since 'tez.ignore.lib.uris' is set to true
2016-05-17 22:18:26,926 INFO client.TezClient (TezCommonUtils.java:createTezSystemStagingPath(118)) - Tez system stage directory file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.5.x-tez-0.6.x/build/test/tmp/.staging/A69585C002A74BF4ABAEEC706FE1A188/A9088A6DF9614808BCA4DF4454CB724E/.tez/application_1463523506869_0001 doesn't exist and is created
2016-05-17 22:18:27,003 INFO client.LocalClient (LocalClient.java:createDAGAppMaster(328)) - DAGAppMaster thread has been created
2016-05-17 22:18:27,003 INFO client.LocalClient (LocalClient.java:startDAGAppMaster(251)) - DAGAppMaster is not created wait for 100ms...
2016-05-17 22:18:27,004 INFO client.LocalClient (LocalClient.java:run(275)) - Using staging directory: /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.5.x-tez-0.6.x/build/test/tmp/.staging/A69585C002A74BF4ABAEEC706FE1A188/A9088A6DF9614808BCA4DF4454CB724E/.tez/application_1463523506869_0001
2016-05-17 22:18:27,034 INFO app.DAGAppMaster (DAGAppMaster.java:<init>(309)) - Created DAGAppMaster for application appattempt_1463523506869_0001_000000, versionInfo=[ component=tez-dag, version=0.6.2, revision=c04283bf91fd96823eec37056d5c8b1a92728265, SCM-URL=scm:git:https://git-wip-us.apache.org/repos/asf/tez.git, buildTime=20150804-1436 ]
2016-05-17 22:18:27,076 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(333)) - Comparing client version with AM version, clientVersion=0.6.2, AMVersion=0.6.2
2016-05-17 22:18:27,091 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(386)) - Adding session token to jobTokenSecretManager for application
2016-05-17 22:18:27,104 INFO client.LocalClient (LocalClient.java:startDAGAppMaster(236)) - DAGAppMaster state: NEW
2016-05-17 22:18:27,104 INFO client.LocalClient (LocalClient.java:startDAGAppMaster(238)) - DAGAppMaster is not started wait for 100ms...
2016-05-17 22:18:27,104 INFO client.LocalClient (LocalClient.java:startDAGAppMaster(251)) - DAGAppMaster is not created wait for 100ms...
2016-05-17 22:18:27,116 INFO node.AMNodeTracker (AMNodeTracker.java:serviceInit(77)) - blacklistDisablePercent is 33, blacklistingEnabled: false, maxTaskFailuresPerNode: 1
2016-05-17 22:18:27,120 INFO history.HistoryEventHandler (HistoryEventHandler.java:serviceInit(50)) - Initializing HistoryEventHandler
2016-05-17 22:18:27,123 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:serviceInit(66)) - Log file location for SimpleHistoryLoggingService not specified, defaulting to containerLogDir=/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.5.x-tez-0.6.x/build/test/tmp/.staging/A69585C002A74BF4ABAEEC706FE1A188/A9088A6DF9614808BCA4DF4454CB724E/.tez/application_1463523506869_0001/localmode-log-dir
2016-05-17 22:18:27,124 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:serviceInit(88)) - Initializing SimpleHistoryLoggingService, logFileLocation=/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.5.x-tez-0.6.x/build/test/tmp/.staging/A69585C002A74BF4ABAEEC706FE1A188/A9088A6DF9614808BCA4DF4454CB724E/.tez/application_1463523506869_0001/localmode-log-dir/history.txt.appattempt_1463523506869_0001_000000, maxErrors=10
2016-05-17 22:18:27,124 INFO recovery.RecoveryService (RecoveryService.java:serviceInit(109)) - Initializing RecoveryService
2016-05-17 22:18:27,126 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:N/A][Event:AM_LAUNCHED]: appAttemptId=appattempt_1463523506869_0001_000000, appSubmitTime=1463523507019, launchTime=1463523507031
2016-05-17 22:18:27,131 INFO history.HistoryEventHandler (HistoryEventHandler.java:serviceStart(73)) - Starting HistoryEventHandler
2016-05-17 22:18:27,132 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:serviceStart(95)) - Starting SimpleHistoryLoggingService
2016-05-17 22:18:27,135 INFO recovery.RecoveryService (RecoveryService.java:serviceStart(127)) - Starting RecoveryService
2016-05-17 22:18:27,137 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event AM_LAUNCHED to history file
2016-05-17 22:18:27,199 INFO client.DAGClientServer (DAGClientServer.java:serviceStart(91)) - Instantiated DAGClientRPCServer at ip-10-123-148-208.ec2.internal/10.123.148.208:48034
2016-05-17 22:18:27,204 INFO client.LocalClient (LocalClient.java:startDAGAppMaster(236)) - DAGAppMaster state: INITED
2016-05-17 22:18:27,205 INFO client.LocalClient (LocalClient.java:startDAGAppMaster(240)) - DAGAppMaster is not startetd wait for 100ms...
2016-05-17 22:18:27,205 INFO client.LocalClient (LocalClient.java:startDAGAppMaster(251)) - DAGAppMaster is not created wait for 100ms...
2016-05-17 22:18:27,240 WARN impl.MetricsConfig (MetricsConfig.java:loadFirst(124)) - Cannot locate configuration: tried hadoop-metrics2-dagappmaster.properties,hadoop-metrics2.properties
2016-05-17 22:18:27,305 INFO client.LocalClient (LocalClient.java:startDAGAppMaster(236)) - DAGAppMaster state: INITED
2016-05-17 22:18:27,305 INFO client.LocalClient (LocalClient.java:startDAGAppMaster(240)) - DAGAppMaster is not startetd wait for 100ms...
2016-05-17 22:18:27,306 INFO client.LocalClient (LocalClient.java:startDAGAppMaster(251)) - DAGAppMaster is not created wait for 100ms...
2016-05-17 22:18:27,332 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event AM_STARTED to history file
2016-05-17 22:18:27,333 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:N/A][Event:AM_STARTED]: appAttemptId=appattempt_1463523506869_0001_000000, startTime=1463523507331
2016-05-17 22:18:27,333 INFO app.DAGAppMaster (DAGAppMaster.java:serviceStart(1644)) - In Session mode. Waiting for DAG over RPC
2016-05-17 22:18:27,406 INFO client.LocalClient (LocalClient.java:startDAGAppMaster(236)) - DAGAppMaster state: IDLE
2016-05-17 22:18:27,422 INFO client.TezClient (TezClient.java:start(367)) - The url to track the Tez Session: N/A
2016-05-17 22:18:27,422 INFO client.TezClient (TezClient.java:submitDAGSession(405)) - Submitting dag to TezSession, sessionName=(1/1) ...stIdentityBuffer/identity, applicationId=application_1463523506869_0001, dagName=[A69585C002A74BF4ABAEEC706FE1A188/A9088A6DF9614808BCA4DF4454CB724E] test/(1/1) ...stIdentityBuffer/identity
2016-05-17 22:18:27,687 INFO app.DAGAppMaster (DAGAppMaster.java:submitDAGToAppMaster(1126)) - Starting DAG submitted via RPC: [A69585C002A74BF4ABAEEC706FE1A188/A9088A6DF9614808BCA4DF4454CB724E] test/(1/1) ...stIdentityBuffer/identity
2016-05-17 22:18:27,806 INFO app.DAGAppMaster (DAGAppMaster.java:generateDAGVizFile(870)) - Generating DAG graphviz file, dagId=dag_1463523506869_0001_1, filePath=/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.5.x-tez-0.6.x/build/test/tmp/.staging/A69585C002A74BF4ABAEEC706FE1A188/A9088A6DF9614808BCA4DF4454CB724E/.tez/application_1463523506869_0001/localmode-log-dir/dag_1463523506869_0001_1.dot
2016-05-17 22:18:27,810 INFO app.DAGAppMaster (DAGAppMaster.java:writePBTextFile(887)) - Writing DAG plan to: /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.5.x-tez-0.6.x/build/test/tmp/.staging/A69585C002A74BF4ABAEEC706FE1A188/A9088A6DF9614808BCA4DF4454CB724E/.tez/application_1463523506869_0001/localmode-log-dir/dag_1463523506869_0001_1-tez-dag.pb.txt
2016-05-17 22:18:27,925 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(164)) - Redirecting log file based on addend: dag_1463523506869_0001_1
2016-05-17 22:18:27,925 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(179)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-17 22:18:27,925 INFO app.DAGAppMaster (DAGAppMaster.java:startDAG(2051)) - Running DAG: [A69585C002A74BF4ABAEEC706FE1A188/A9088A6DF9614808BCA4DF4454CB724E] test/(1/1) ...stIdentityBuffer/identity
2016-05-17 22:18:27 Running Dag: dag_1463523506869_0001_1
2016-05-17 22:18:27,953 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event DAG_SUBMITTED to history file
2016-05-17 22:18:27,953 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1463523506869_0001_1][Event:DAG_SUBMITTED]: dagID=dag_1463523506869_0001_1, submitTime=1463523507688
2016-05-17 22:18:27,981 INFO impl.VertexImpl (VertexImpl.java:setAdditionalOutputs(4033)) - setting additional outputs for vertex E7D2E5C67A994389B15E179A53A02836
2016-05-17 22:18:27,988 INFO impl.DAGImpl (DAGImpl.java:assignDAGScheduler(1381)) - Using DAG Scheduler: org.apache.tez.dag.app.dag.impl.DAGSchedulerNaturalOrderControlled
2016-05-17 22:18:27,990 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event DAG_INITIALIZED to history file
2016-05-17 22:18:27,994 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1463523506869_0001_1][Event:DAG_INITIALIZED]: dagID=dag_1463523506869_0001_1, initTime=1463523507954
2016-05-17 22:18:27,995 INFO impl.DAGImpl (DAGImpl.java:handle(973)) - dag_1463523506869_0001_1 transitioned from NEW to INITED due to event DAG_INIT
2016-05-17 22:18:27,997 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event DAG_STARTED to history file
2016-05-17 22:18:27,997 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1463523506869_0001_1][Event:DAG_STARTED]: dagID=dag_1463523506869_0001_1, startTime=1463523507996
2016-05-17 22:18:27,997 INFO impl.DAGImpl (DAGImpl.java:transition(1614)) - Added additional resources : [[]] to classpath
2016-05-17 22:18:27,998 INFO impl.DAGImpl (DAGImpl.java:handle(973)) - dag_1463523506869_0001_1 transitioned from INITED to RUNNING due to event DAG_START
2016-05-17 22:18:27,999 INFO impl.VertexImpl (VertexImpl.java:setupVertex(2068)) - Root Inputs exist for Vertex: E205BB9BAA7744F9B0E4DC91DE388B4F : {D3D291A4059F47829ACEC1E10758D0E1={InputName=D3D291A4059F47829ACEC1E10758D0E1}, {Descriptor=ClassName=org.apache.tez.mapreduce.input.MRInput, hasPayload=true}, {ControllerDescriptor=ClassName=org.apache.tez.mapreduce.common.MRInputAMSplitGenerator, hasPayload=false}}
2016-05-17 22:18:27,999 INFO impl.VertexImpl (VertexImpl.java:setupVertex(2078)) - Starting root input initializer for input: D3D291A4059F47829ACEC1E10758D0E1, with class: [org.apache.tez.mapreduce.common.MRInputAMSplitGenerator]
2016-05-17 22:18:27,999 INFO impl.VertexImpl (VertexImpl.java:assignVertexManager(2191)) - Setting vertexManager to RootInputVertexManager for vertex_1463523506869_0001_1_00 [E205BB9BAA7744F9B0E4DC91DE388B4F]
2016-05-17 22:18:28,007 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(2983)) - Num tasks is -1. Expecting VertexManager/InputInitializers/1-1 split to set #tasks for the vertex vertex_1463523506869_0001_1_00 [E205BB9BAA7744F9B0E4DC91DE388B4F]
2016-05-17 22:18:28,007 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(2987)) - Vertex will initialize from input initializer. vertex_1463523506869_0001_1_00 [E205BB9BAA7744F9B0E4DC91DE388B4F]
2016-05-17 22:18:28,009 INFO impl.VertexImpl (VertexImpl.java:setupInputInitializerManager(3933)) - Vertex will initialize via inputInitializers vertex_1463523506869_0001_1_00 [E205BB9BAA7744F9B0E4DC91DE388B4F]. Starting root input initializers: 1
2016-05-17 22:18:28,012 INFO client.TezClient (TezClient.java:submitDAGSession(478)) - Submitted dag to TezSession, sessionName=(1/1) ...stIdentityBuffer/identity, applicationId=application_1463523506869_0001, dagName=[A69585C002A74BF4ABAEEC706FE1A188/A9088A6DF9614808BCA4DF4454CB724E] test/(1/1) ...stIdentityBuffer/identity
2016-05-17 22:18:28,014 INFO dag.RootInputInitializerManager (RootInputInitializerManager.java:run(242)) - Starting InputInitializer for Input: D3D291A4059F47829ACEC1E10758D0E1 on vertex vertex_1463523506869_0001_1_00 [E205BB9BAA7744F9B0E4DC91DE388B4F]
2016-05-17 22:18:28,031 INFO common.MRInputAMSplitGenerator (MRInputAMSplitGenerator.java:initialize(92)) - Emitting serialized splits: true
2016-05-17 22:18:28,033 INFO common.MRInputAMSplitGenerator (MRInputAMSplitGenerator.java:initialize(110)) - Input D3D291A4059F47829ACEC1E10758D0E1 asking for 1 tasks. Headroom: 672 Task Resource: 1024 waves: 1.7
2016-05-17 22:18:28,034 INFO common.MRInputAMSplitGenerator (MRInputAMSplitGenerator.java:initialize(121)) - Grouping input splits
2016-05-17 22:18:28,035 INFO hadoop.MRInputHelpers (MRInputHelpers.java:generateInputSplitsToMem(294)) - Generating mapred api input splits
2016-05-17 22:18:28,046 INFO impl.VertexImpl (VertexImpl.java:handle(1564)) - vertex_1463523506869_0001_1_00 [E205BB9BAA7744F9B0E4DC91DE388B4F] transitioned from NEW to INITIALIZING due to event V_INIT
2016-05-17 22:18:28,048 INFO impl.VertexImpl (VertexImpl.java:assignVertexManager(2203)) - Setting vertexManager to ShuffleVertexManager for vertex_1463523506869_0001_1_01 [E7D2E5C67A994389B15E179A53A02836]
2016-05-17 22:18:28,075 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [test] submitted tez dag to app master: application_1463523506869_0001, with dag id: dag_1463523506869_0001_1
2016-05-17 22:18:28,085 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:initialize(696)) - Shuffle Vertex Manager: settings minFrac:0.25 maxFrac:0.75 auto:false desiredTaskIput:104857600 minTasks:1
2016-05-17 22:18:28,086 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(3014)) - Creating 1 tasks for vertex: vertex_1463523506869_0001_1_01 [E7D2E5C67A994389B15E179A53A02836]
2016-05-17 22:18:28,089 INFO hadoop.MRInputHelpers (MRInputHelpers.java:generateInputSplitsToMem(300)) - NumSplits: 1, SerializedSize: 396
2016-05-17 22:18:28,092 INFO dag.RootInputInitializerManager (RootInputInitializerManager.java:onSuccess(272)) - Succeeded InputInitializer for Input: D3D291A4059F47829ACEC1E10758D0E1 on vertex vertex_1463523506869_0001_1_00 [E205BB9BAA7744F9B0E4DC91DE388B4F]
2016-05-17 22:18:28,100 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(3026)) - Directly initializing vertex: vertex_1463523506869_0001_1_01 [E7D2E5C67A994389B15E179A53A02836]
2016-05-17 22:18:28,102 INFO impl.VertexImpl (VertexImpl.java:initializeCommitters(1915)) - Invoking committer inits for vertex, vertexId=vertex_1463523506869_0001_1_01 [E7D2E5C67A994389B15E179A53A02836]
2016-05-17 22:18:28,102 INFO impl.VertexImpl (VertexImpl.java:initializeCommitters(1927)) - Instantiating committer for output=D856B73605D74B1A813E6470D8E6523D, vertexId=vertex_1463523506869_0001_1_01 [E7D2E5C67A994389B15E179A53A02836], committerClass=org.apache.tez.mapreduce.committer.MROutputCommitter
2016-05-17 22:18:28,105 INFO impl.VertexImpl (VertexImpl.java:run(1945)) - Invoking committer init for output=D856B73605D74B1A813E6470D8E6523D, vertexId=vertex_1463523506869_0001_1_01 [E7D2E5C67A994389B15E179A53A02836]
2016-05-17 22:18:28,116 INFO committer.MROutputCommitter (MROutputCommitter.java:getOutputCommitter(145)) - OutputCommitter for outputName=D856B73605D74B1A813E6470D8E6523D, vertexName=E7D2E5C67A994389B15E179A53A02836, outputCommitterClass=org.apache.hadoop.mapred.FileOutputCommitter
2016-05-17 22:18:28,125 INFO impl.VertexImpl (VertexImpl.java:run(1949)) - Invoking committer setup for output=D856B73605D74B1A813E6470D8E6523D, vertexId=vertex_1463523506869_0001_1_01 [E7D2E5C67A994389B15E179A53A02836]
2016-05-17 22:18:28,129 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1463523506869_0001_1][Event:VERTEX_INITIALIZED]: vertexName=E7D2E5C67A994389B15E179A53A02836, vertexId=vertex_1463523506869_0001_1_01, initRequestedTime=1463523508046, initedTime=1463523508127, numTasks=1, processorName=cascading.flow.tez.FlowProcessor, additionalInputsCount=0
2016-05-17 22:18:28,129 INFO impl.VertexImpl (VertexImpl.java:handle(1564)) - vertex_1463523506869_0001_1_01 [E7D2E5C67A994389B15E179A53A02836] transitioned from NEW to INITED due to event V_INIT
2016-05-17 22:18:28,131 INFO impl.VertexImpl (VertexImpl.java:canInitVertex(3195)) - Cannot init vertex: vertex_1463523506869_0001_1_00 [E205BB9BAA7744F9B0E4DC91DE388B4F] numTasks: -1 numUnitializedEdges: 0 numInitializedInputs: 0 initWaitsForRootInitializers: true
2016-05-17 22:18:28,131 INFO impl.VertexImpl (VertexImpl.java:setParallelism(1374)) - Got updated RootInputsSpecs: {D3D291A4059F47829ACEC1E10758D0E1=forAllWorkUnits=true, update=[1]}
2016-05-17 22:18:28,133 INFO impl.VertexImpl (VertexImpl.java:setParallelism(1393)) - Vertex vertex_1463523506869_0001_1_00 [E205BB9BAA7744F9B0E4DC91DE388B4F] parallelism set to 1
2016-05-17 22:18:28,133 INFO impl.VertexImpl (VertexImpl.java:canInitVertex(3195)) - Cannot init vertex: vertex_1463523506869_0001_1_00 [E205BB9BAA7744F9B0E4DC91DE388B4F] numTasks: 1 numUnitializedEdges: 0 numInitializedInputs: 0 initWaitsForRootInitializers: true
2016-05-17 22:18:28,134 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event VERTEX_INITIALIZED to history file
2016-05-17 22:18:28,147 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1463523506869_0001_1][Event:VERTEX_DATA_MOVEMENT_EVENTS_GENERATED]: vertexId=vertex_1463523506869_0001_1_00, eventCount=1
2016-05-17 22:18:28,149 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event VERTEX_INITIALIZED to history file
2016-05-17 22:18:28,150 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1463523506869_0001_1][Event:VERTEX_INITIALIZED]: vertexName=E205BB9BAA7744F9B0E4DC91DE388B4F, vertexId=vertex_1463523506869_0001_1_00, initRequestedTime=1463523507999, initedTime=1463523508148, numTasks=1, processorName=cascading.flow.tez.FlowProcessor, additionalInputsCount=1
2016-05-17 22:18:28,151 INFO impl.VertexImpl (VertexImpl.java:startIfPossible(3061)) - Triggering start event for vertex: vertex_1463523506869_0001_1_00 [E205BB9BAA7744F9B0E4DC91DE388B4F] with distanceFromRoot: 0
2016-05-17 22:18:28,152 INFO impl.VertexImpl (VertexImpl.java:handle(1564)) - vertex_1463523506869_0001_1_00 [E205BB9BAA7744F9B0E4DC91DE388B4F] transitioned from INITIALIZING to INITED due to event V_ROOT_INPUT_INITIALIZED
2016-05-17 22:18:28,154 INFO impl.ImmediateStartVertexManager (ImmediateStartVertexManager.java:scheduleTasks(99)) - Starting 1 in E205BB9BAA7744F9B0E4DC91DE388B4F
2016-05-17 22:18:28,154 INFO impl.VertexImpl (VertexImpl.java:scheduleTasks(1250)) - Routing pending task events for vertex: vertex_1463523506869_0001_1_00 [E205BB9BAA7744F9B0E4DC91DE388B4F]
2016-05-17 22:18:28,157 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event VERTEX_STARTED to history file
2016-05-17 22:18:28,158 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1463523506869_0001_1][Event:VERTEX_STARTED]: vertexId=vertex_1463523506869_0001_1_00, startRequestedTime=1463523508152, startedTime=1463523508153
2016-05-17 22:18:28,160 INFO impl.VertexImpl (VertexImpl.java:handle(1564)) - vertex_1463523506869_0001_1_00 [E205BB9BAA7744F9B0E4DC91DE388B4F] transitioned from INITED to RUNNING due to event V_START
2016-05-17 22:18:28,198 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event TASK_STARTED to history file
2016-05-17 22:18:28,199 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1463523506869_0001_1][Event:TASK_STARTED]: vertexName=E205BB9BAA7744F9B0E4DC91DE388B4F, taskId=task_1463523506869_0001_1_00_000000, scheduledTime=1463523508195, launchTime=1463523508195
2016-05-17 22:18:28,201 INFO impl.TaskImpl (TaskImpl.java:handle(862)) - task_1463523506869_0001_1_00_000000 Task Transitioned from NEW to SCHEDULED due to event T_SCHEDULE
2016-05-17 22:18:28,202 INFO impl.VertexImpl (VertexImpl.java:transition(3155)) - Source vertex started: vertex_1463523506869_0001_1_00 for vertex: vertex_1463523506869_0001_1_01 [E7D2E5C67A994389B15E179A53A02836] numStartedSources: 1 numSources: 1
2016-05-17 22:18:28,203 INFO impl.VertexImpl (VertexImpl.java:startIfPossible(3061)) - Triggering start event for vertex: vertex_1463523506869_0001_1_01 [E7D2E5C67A994389B15E179A53A02836] with distanceFromRoot: 1
2016-05-17 22:18:28,206 INFO impl.DAGSchedulerNaturalOrderControlled (DAGSchedulerNaturalOrderControlled.java:scheduleTask(111)) - Scheduled vertex: vertex_1463523506869_0001_1_00 [E205BB9BAA7744F9B0E4DC91DE388B4F]
2016-05-17 22:18:28,215 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:handleVertexStateUpdate(720)) - Received configured notification : CONFIGURED for vertex: E205BB9BAA7744F9B0E4DC91DE388B4F in vertex: E7D2E5C67A994389B15E179A53A02836
2016-05-17 22:18:28,218 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:onVertexStarted(355)) - OnVertexStarted vertex: E7D2E5C67A994389B15E179A53A02836 with 1 source tasks and 1 pending tasks
2016-05-17 22:18:28,219 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event VERTEX_STARTED to history file
2016-05-17 22:18:28,220 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1463523506869_0001_1][Event:VERTEX_STARTED]: vertexId=vertex_1463523506869_0001_1_01, startRequestedTime=1463523508207, startedTime=1463523508207
2016-05-17 22:18:28,221 INFO impl.VertexImpl (VertexImpl.java:handle(1564)) - vertex_1463523506869_0001_1_01 [E7D2E5C67A994389B15E179A53A02836] transitioned from INITED to RUNNING due to event V_START
2016-05-17 22:18:28,224 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:transition(1014)) - remoteTaskSpec:DAGName : [A69585C002A74BF4ABAEEC706FE1A188/A9088A6DF9614808BCA4DF4454CB724E] test/(1/1) ...stIdentityBuffer/identity, VertexName: E205BB9BAA7744F9B0E4DC91DE388B4F, VertexParallelism: 1, TaskAttemptID:attempt_1463523506869_0001_1_00_000000_0, processorName=cascading.flow.tez.FlowProcessor, inputSpecListSize=1, outputSpecListSize=1, inputSpecList=[{{ sourceVertexName=D3D291A4059F47829ACEC1E10758D0E1, physicalEdgeCount=1, inputClassName=org.apache.tez.mapreduce.input.MRInput }}, ], outputSpecList=[{{ destinationVertexName=E7D2E5C67A994389B15E179A53A02836, physicalEdgeCount=1, outputClassName=org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput }}, ]
2016-05-17 22:18:28,226 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:handle(692)) - attempt_1463523506869_0001_1_00_000000_0 TaskAttempt Transitioned from NEW to START_WAIT due to event TA_SCHEDULE
2016-05-17 22:18:28,227 INFO rm.TaskSchedulerEventHandler (TaskSchedulerEventHandler.java:handleEvent(149)) - Processing the event EventType: S_TA_LAUNCH_REQUEST
2016-05-17 22:18:28,265 INFO node.AMNodeTracker (AMNodeTracker.java:nodeSeen(91)) - Adding new node: 127.0.0.1:0
2016-05-17 22:18:28,268 INFO container.AMContainerHelpers (AMContainerHelpers.java:createCommonContainerLaunchContext(112)) - Adding #1 tokens and #0 secret keys for NM use for launching container
2016-05-17 22:18:28,269 INFO container.AMContainerHelpers (AMContainerHelpers.java:createCommonContainerLaunchContext(122)) - Putting shuffle token in serviceData
2016-05-17 22:18:28,272 INFO container.AMContainerImpl (AMContainerImpl.java:transition(521)) - Sending Launch Request for Container with id: container_1463523506869_0001_00_000001
2016-05-17 22:18:28,272 INFO container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1463523506869_0001_00_000001 transitioned from ALLOCATED to LAUNCHING via event C_LAUNCH_REQUEST
2016-05-17 22:18:28,308 INFO task.TezChild (TezChild.java:newTezChild(432)) - PID, containerIdentifier: null, container_1463523506869_0001_00_000001
2016-05-17 22:18:28,309 WARN impl.MetricsSystemImpl (MetricsSystemImpl.java:init(145)) - DAGAppMaster metrics system already initialized!
2016-05-17 22:18:28,325 INFO container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1463523506869_0001_00_000001 transitioned from LAUNCHING to IDLE via event C_LAUNCHED
2016-05-17 22:18:28,326 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event CONTAINER_LAUNCHED to history file
2016-05-17 22:18:28,327 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1463523506869_0001_1][Event:CONTAINER_LAUNCHED]: containerId=container_1463523506869_0001_00_000001, launchTime=1463523508326
2016-05-17 22:18:28,330 INFO task.ContainerReporter (ContainerReporter.java:call(55)) - Attempting to fetch new task for container container_1463523506869_0001_00_000001
2016-05-17 22:18:28,334 INFO container.AMContainerImpl (AMContainerImpl.java:transition(827)) - Assigned taskAttempt + [attempt_1463523506869_0001_1_00_000000_0] to container: [container_1463523506869_0001_00_000001]
2016-05-17 22:18:28,334 INFO container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1463523506869_0001_00_000001 transitioned from IDLE to RUNNING via event C_PULL_TA
2016-05-17 22:18:28,335 INFO app.TaskAttemptListenerImpTezDag (TaskAttemptListenerImpTezDag.java:getTask(240)) - Container with id: container_1463523506869_0001_00_000001 given task: attempt_1463523506869_0001_1_00_000000_0
2016-05-17 22:18:28,337 INFO task.ContainerReporter (ContainerReporter.java:call(65)) - Got TaskUpdate for containerId= container_1463523506869_0001_00_000001: 1 ms after starting to poll. TaskInfo: shouldDie: false, currentTaskAttemptId: attempt_1463523506869_0001_1_00_000000_0
2016-05-17 22:18:28,337 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:transition(1146)) - TaskAttempt: [attempt_1463523506869_0001_1_00_000000_0] started. Is using containerId: [container_1463523506869_0001_00_000001] on NM: [127.0.0.1:0]
2016-05-17 22:18:28 Starting to run new task attempt: attempt_1463523506869_0001_1_00_000000_0
2016-05-17 22:18:28,339 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1463523506869_0001_1][Event:TASK_ATTEMPT_STARTED]: vertexName=E205BB9BAA7744F9B0E4DC91DE388B4F, taskAttemptId=attempt_1463523506869_0001_1_00_000000_0, startTime=1463523508335, containerId=container_1463523506869_0001_00_000001, nodeId=127.0.0.1:0, inProgressLogs=127.0.0.1:0/node/containerlogs/container_1463523506869_0001_00_000001/teamcity, completedLogs=
2016-05-17 22:18:28,339 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(164)) - Redirecting log file based on addend: attempt_1463523506869_0001_1_00_000000_0
2016-05-17 22:18:28,340 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(179)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-17 22:18:28,340 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:handle(692)) - attempt_1463523506869_0001_1_00_000000_0 TaskAttempt Transitioned from START_WAIT to RUNNING due to event TA_STARTED_REMOTELY
2016-05-17 22:18:28,340 INFO counters.Limits (Limits.java:ensureInitialized(59)) - Counter limits initialized with parameters: GROUP_NAME_MAX=128, MAX_GROUPS=500, COUNTER_NAME_MAX=64, MAX_COUNTERS=1200
2016-05-17 22:18:28,341 INFO task.TezChild (TezChild.java:handleNewTaskCredentials(273)) - Refreshing UGI since Credentials have changed
2016-05-17 22:18:28,341 INFO task.TezChild (TezChild.java:handleNewTaskCredentials(276)) - Credentials : #Tokens=1, #SecretKeys=0
2016-05-17 22:18:28,341 INFO task.TezChild (TezChild.java:handleNewTaskLocalResources(302)) - Localizing additional local resources for Task : {}
2016-05-17 22:18:28,342 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event TASK_ATTEMPT_STARTED to history file
2016-05-17 22:18:28,343 INFO task.TezChild (TezChild.java:handleNewTaskLocalResources(312)) - Done localizing additional resources
2016-05-17 22:18:28,347 INFO impl.TaskImpl (TaskImpl.java:handle(862)) - task_1463523506869_0001_1_00_000000 Task Transitioned from SCHEDULED to RUNNING due to event T_ATTEMPT_LAUNCHED
2016-05-17 22:18:28,368 INFO metrics.TaskCounterUpdater (TaskCounterUpdater.java:initResourceCalculatorPlugin(148)) - Using ResourceCalculatorProcessTree : [ ]
2016-05-17 22:18:28,368 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:<init>(149)) - Initializing LogicalIOProcessorRuntimeTask with TaskSpec: DAGName : [A69585C002A74BF4ABAEEC706FE1A188/A9088A6DF9614808BCA4DF4454CB724E] test/(1/1) ...stIdentityBuffer/identity, VertexName: E205BB9BAA7744F9B0E4DC91DE388B4F, VertexParallelism: 1, TaskAttemptID:attempt_1463523506869_0001_1_00_000000_0, processorName=cascading.flow.tez.FlowProcessor, inputSpecListSize=1, outputSpecListSize=1, inputSpecList=[{{ sourceVertexName=D3D291A4059F47829ACEC1E10758D0E1, physicalEdgeCount=1, inputClassName=org.apache.tez.mapreduce.input.MRInput }}, ], outputSpecList=[{{ destinationVertexName=E7D2E5C67A994389B15E179A53A02836, physicalEdgeCount=1, outputClassName=org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput }}, ]
2016-05-17 22:18:28,371 INFO resources.MemoryDistributor (MemoryDistributor.java:<init>(85)) - InitialMemoryDistributor (isEnabled=true) invoked with: numInputs=1, numOutputs=1, JVM.maxFree=704643072
2016-05-17 22:18:28,375 INFO task.TezTaskRunner (TezTaskRunner.java:run(172)) - Initializing task, taskAttemptId=attempt_1463523506869_0001_1_00_000000_0
2016-05-17 22:18:28,375 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(191)) - Initializing LogicalProcessorIORuntimeTask
2016-05-17 22:18:28,375 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(195)) - Creating processor, processorClassName=cascading.flow.tez.FlowProcessor
2016-05-17 22:18:28,379 INFO impl.TaskImpl (TaskImpl.java:getTaskAttemptTezEvents(481)) - TaskAttempt:attempt_1463523506869_0001_1_00_000000_0 sent events: (0-1)
2016-05-17 22:18:28,383 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(389)) - Initializing Input using InputSpec: { sourceVertexName=D3D291A4059F47829ACEC1E10758D0E1, physicalEdgeCount=1, inputClassName=org.apache.tez.mapreduce.input.MRInput }
2016-05-17 22:18:28,384 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initializeLogicalIOProcessor(488)) - Initializing processor, processorClassName=cascading.flow.tez.FlowProcessor
2016-05-17 22:18:28,384 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(438)) - Initializing Output using OutputSpec: { destinationVertexName=E7D2E5C67A994389B15E179A53A02836, physicalEdgeCount=1, outputClassName=org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput }
2016-05-17 22:18:28,385 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:createInput(537)) - Creating Input
2016-05-17 22:18:28,386 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:createOutput(565)) - Creating Output
2016-05-17 22:18:28,386 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(397)) - Initializing Input with src edge: D3D291A4059F47829ACEC1E10758D0E1
2016-05-17 22:18:28,390 INFO resources.MemoryDistributor$RequestorInfo (MemoryDistributor.java:<init>(233)) - Received request: 0, type: INPUT, componentVertexName: D3D291A4059F47829ACEC1E10758D0E1
2016-05-17 22:18:28,396 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(446)) - Initializing Output with dest edge: E7D2E5C67A994389B15E179A53A02836
2016-05-17 22:18:28,412 INFO tez.FlowProcessor (FlowProcessor.java:initialize(88)) - cascading version: 3.1.0
2016-05-17 22:18:28,413 INFO input.MRInput (MRInput.java:initialize(440)) - Using New mapreduce API: false, split information via event: true
2016-05-17 22:18:28,414 INFO input.MRInput (MRInput.java:initializeInternal(493)) - Initialzed MRInput: D3D291A4059F47829ACEC1E10758D0E1
2016-05-17 22:18:28,414 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(403)) - Initialized Input with src edge: D3D291A4059F47829ACEC1E10758D0E1
2016-05-17 22:18:28,415 INFO impl.ExternalSorter (ExternalSorter.java:getInitialMemoryRequirement(293)) - Requested SortBufferSize (tez.runtime.io.sort.mb): 100
2016-05-17 22:18:28,416 INFO resources.MemoryDistributor$RequestorInfo (MemoryDistributor.java:<init>(233)) - Received request: 104857600, type: OUTPUT, componentVertexName: E7D2E5C67A994389B15E179A53A02836
2016-05-17 22:18:28,417 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(452)) - Initialized Output with dest edge: E7D2E5C67A994389B15E179A53A02836
2016-05-17 22:18:28,429 INFO tez.FlowProcessor (FlowProcessor.java:initialize(94)) - flow node id: E205BB9BAA7744F9B0E4DC91DE388B4F, ordinal: 0
2016-05-17 22:18:28,429 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: E205BB9BAA7744F9B0E4DC91DE388B4F, mem on start (mb), free: 255, total: 281, max: 672
2016-05-17 22:18:28,430 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initializeLogicalIOProcessor(491)) - Initialized processor, processorClassName=cascading.flow.tez.FlowProcessor
2016-05-17 22:18:28,430 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(220)) - Waiting for 2 initializers to finish
2016-05-17 22:18:28,430 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(220)) - Waiting for 1 initializers to finish
2016-05-17 22:18:28,430 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(233)) - All initializers finished
2016-05-17 22:18:28,431 INFO resources.MemoryDistributor (MemoryDistributor.java:makeInitialAllocations(124)) - Using Allocator class: org.apache.tez.runtime.library.resources.WeightedScalingMemoryDistributor
2016-05-17 22:18:28,437 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:getRequestTypeForClass(197)) - Falling back to RequestType.OTHER for class: org.apache.tez.mapreduce.input.MRInput
2016-05-17 22:18:28,438 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:initialProcessMemoryRequestContext(171)) - ScaleFactor: 1, for type: OTHER
2016-05-17 22:18:28,438 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:initialProcessMemoryRequestContext(171)) - ScaleFactor: 12, for type: SORTED_OUTPUT
2016-05-17 22:18:28,438 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:computeReservedFraction(262)) - InitialReservationFraction=0.3, AdditionalReservationFractionForIOs=0.03, finalReserveFractionUsed=0.32999999999999996
2016-05-17 22:18:28,438 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:assignMemory(126)) - Scaling Requests. NumRequests: 2, numScaledRequests: 13, TotalRequested: 104857600, TotalRequestedScaled: 9.679163076923077E7, TotalJVMHeap: 704643072, TotalAvailable: 472110858, TotalRequested/TotalJVMHeap:0.15
2016-05-17 22:18:28,439 INFO resources.MemoryDistributor (MemoryDistributor.java:makeInitialAllocations(138)) - Informing: INPUT, D3D291A4059F47829ACEC1E10758D0E1, org.apache.tez.mapreduce.input.MRInput: requested=0, allocated=0
2016-05-17 22:18:28,439 INFO resources.MemoryDistributor (MemoryDistributor.java:makeInitialAllocations(138)) - Informing: OUTPUT, E7D2E5C67A994389B15E179A53A02836, org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput: requested=104857600, allocated=104857600
2016-05-17 22:18:28,439 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(255)) - Starting Inputs/Outputs
2016-05-17 22:18:28,440 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(268)) - Input: D3D291A4059F47829ACEC1E10758D0E1 being auto started by the framework. Subsequent instances will not be auto-started
2016-05-17 22:18:28,440 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(289)) - Num IOs determined for AutoStart: 1
2016-05-17 22:18:28,440 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(291)) - Waiting for 1 IOs to start
2016-05-17 22:18:28,442 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(419)) - Starting Input with src edge: D3D291A4059F47829ACEC1E10758D0E1
2016-05-17 22:18:28,442 INFO input.MRInput (MRInput.java:start(450)) - MRInput setup to received events1
2016-05-17 22:18:28,443 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(421)) - Started Input with src edge: D3D291A4059F47829ACEC1E10758D0E1
2016-05-17 22:18:28,444 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(304)) - AutoStartComplete
2016-05-17 22:18:28,447 INFO task.TezTaskRunner (TezTaskRunner.java:run(175)) - Running task, taskAttemptId=attempt_1463523506869_0001_1_00_000000_0
2016-05-17 22:18:28,449 INFO input.MRInput (MRInput.java:initFromEventInternal(619)) - Initializing RecordReader from event
2016-05-17 22:18:28,457 INFO input.MRInput (MRInput.java:initFromEventInternal(630)) - Split Details -> SplitClass: org.apache.hadoop.mapred.split.TezGroupedSplit, OldSplit: org.apache.hadoop.mapred.split.TezGroupedSplit@7bb9d080
2016-05-17 22:18:28,467 INFO lib.MRReaderMapred (MRReaderMapred.java:setIncrementalConfigParams(163)) - Processing split: org.apache.hadoop.mapred.split.TezGroupedSplit@7bb9d080
2016-05-17 22:18:28,467 INFO input.MRInput (MRInput.java:initFromEventInternal(634)) - Initialized RecordReader from event
2016-05-17 22:18:28,468 INFO input.MRInput (MRInput.java:processSplitEvent(586)) - Notifying on RecordReader Initialized
2016-05-17 22:18:28,479 INFO graph.Hadoop2TezStreamGraph (Hadoop2TezStreamGraph.java:buildGraph(151)) - using streamed source: Hfs["TextLine[['offset', 'line' | long, String]->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.5.x-tez-0.6.x/build/classes/test/data/lhs.txt"]
2016-05-17 22:18:28,511 INFO tez.FlowProcessor (FlowProcessor.java:run(121)) - sourcing from: Hfs["TextLine[['offset', 'line' | long, String]->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.5.x-tez-0.6.x/build/classes/test/data/lhs.txt"] streamed: true, id: D3D291A4059F47829ACEC1E10758D0E1
2016-05-17 22:18:28,511 INFO tez.FlowProcessor (FlowProcessor.java:run(124)) - sinking to: GroupBy(test)[by:[{1}:'num']], id: 76E96E85058B485EB8C48862CCECFE63
2016-05-17 22:18:28,512 INFO element.TezGroupGate (TezGroupGate.java:prepare(98)) - calling OrderedPartitionedKVOutput#start() on: GroupBy(test)[by:[{1}:'num']] 76E96E85058B485EB8C48862CCECFE63
2016-05-17 22:18:28,546 INFO impl.ExternalSorter (ExternalSorter.java:<init>(167)) - keySerializer=cascading.tuple.hadoop.io.KeyTupleSerializer@7eefba6d; valueSerializer=cascading.tuple.hadoop.io.ValueTupleSerializer@2cb88b36; comparator=cascading.tuple.hadoop.util.TupleComparator@2e4b345d; conf=cascading.tuple.hadoop.TupleSerialization,org.apache.hadoop.io.serializer.WritableSerialization,org.apache.hadoop.io.serializer.avro.AvroSpecificSerialization,org.apache.hadoop.io.serializer.avro.AvroReflectSerialization
2016-05-17 22:18:28,549 INFO impl.ExternalSorter (ExternalSorter.java:<init>(224)) - Instantiating Partitioner: [cascading.tuple.tez.util.TuplePartitioner]
2016-05-17 22:18:28,552 INFO common.TezRuntimeUtils (TezRuntimeUtils.java:instantiatePartitioner(109)) - Using partitioner class: cascading.tuple.tez.util.TuplePartitioner
2016-05-17 22:18:28,552 INFO common.TezRuntimeUtils (TezRuntimeUtils.java:instantiateCombiner(65)) - No combiner specified via tez.runtime.combiner.class. Combiner will not be used
2016-05-17 22:18:28,669 INFO dflt.DefaultSorter (DefaultSorter.java:setEquator(328)) - (EQUATOR) 0 kvi 26214396(104857584)
2016-05-17 22:18:28,670 INFO dflt.DefaultSorter (DefaultSorter.java:<init>(150)) - tez.runtime.io.sort.mb: 100
2016-05-17 22:18:28,670 INFO dflt.DefaultSorter (DefaultSorter.java:<init>(151)) - soft limit at 83886080
2016-05-17 22:18:28,670 INFO dflt.DefaultSorter (DefaultSorter.java:<init>(152)) - bufstart = 0; bufvoid = 104857600
2016-05-17 22:18:28,670 INFO dflt.DefaultSorter (DefaultSorter.java:<init>(153)) - kvstart = 26214396; length = 6553600
2016-05-17 22:18:28,672 INFO element.TezSourceStage (TezSourceStage.java:prepare(63)) - calling MRInput#start() on: Hfs["TextLine[['offset', 'line' | long, String]->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.5.x-tez-0.6.x/build/classes/test/data/lhs.txt"]
2016-05-17 22:18:28,673 INFO input.MRInput (MRInput.java:start(450)) - MRInput setup to received events1
2016-05-17 22:18:28,675 INFO tez.FlowProcessor (FlowProcessor.java:waitForInputsReady(202)) - flow node id: E205BB9BAA7744F9B0E4DC91DE388B4F, all 1 inputs ready in: 00:00:00.002
2016-05-17 22:18:28,684 INFO task.TezTaskRunner (TezTaskRunner.java:run(177)) - Closing task, taskAttemptId=attempt_1463523506869_0001_1_00_000000_0
2016-05-17 22:18:28,684 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: E205BB9BAA7744F9B0E4DC91DE388B4F, mem on close (mb), free: 144, total: 281, max: 672
2016-05-17 22:18:28,685 INFO dflt.DefaultSorter (DefaultSorter.java:flush(589)) - Starting flush of map output
2016-05-17 22:18:28,685 INFO dflt.DefaultSorter (DefaultSorter.java:flush(607)) - Sorting & Spilling map output
2016-05-17 22:18:28,685 INFO dflt.DefaultSorter (DefaultSorter.java:flush(608)) - bufstart = 0; bufend = 195; bufvoid = 104857600
2016-05-17 22:18:28,685 INFO dflt.DefaultSorter (DefaultSorter.java:flush(610)) - kvstart = 26214396(104857584); kvend = 26214348(104857392); length = 49/6553600
2016-05-17 22:18:28,702 INFO dflt.DefaultSorter (DefaultSorter.java:spill(829)) - Finished spill 0
2016-05-17 22:18:28 Completed running task attempt: attempt_1463523506869_0001_1_00_000000_0
2016-05-17 22:18:28,723 INFO task.TezTaskRunner (TezTaskRunner.java:run(181)) - Task completed, taskAttemptId=attempt_1463523506869_0001_1_00_000000_0, fatalErrorOccurred=false
2016-05-17 22:18:28,724 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1463523506869_0001_1][Event:VERTEX_DATA_MOVEMENT_EVENTS_GENERATED]: vertexId=vertex_1463523506869_0001_1_00, eventCount=1
2016-05-17 22:18:28,727 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:cleanup(712)) - Final Counters : Counters: 27 [[File System Counters FILE_BYTES_READ=110, FILE_BYTES_WRITTEN=727, FILE_READ_OPS=0, FILE_LARGE_READ_OPS=0, FILE_WRITE_OPS=0][org.apache.tez.common.counters.TaskCounter SPILLED_RECORDS=13, GC_TIME_MILLIS=0, CPU_MILLISECONDS=0, PHYSICAL_MEMORY_BYTES=0, VIRTUAL_MEMORY_BYTES=0, COMMITTED_HEAP_BYTES=295174144, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=13, OUTPUT_BYTES=195, OUTPUT_BYTES_WITH_OVERHEAD=171, OUTPUT_BYTES_PHYSICAL=175, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=0, ADDITIONAL_SPILL_COUNT=0][cascading.flow.SliceCounters Process_Begin_Time=1463523508675, Process_Duration=9, Process_End_Time=1463523508684, Read_Duration=1, Tuples_Read=13, Tuples_Written=13, Write_Duration=1][cascading.flow.StepCounters Tuples_Read=13]]
2016-05-17 22:18:28,727 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event TASK_ATTEMPT_FINISHED to history file
2016-05-17 22:18:28,729 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(164)) - Redirecting log file based on addend:
2016-05-17 22:18:28,730 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(179)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-17 22:18:28,731 INFO task.ContainerReporter (ContainerReporter.java:call(55)) - Attempting to fetch new task for container container_1463523506869_0001_00_000001
2016-05-17 22:18:28,735 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1463523506869_0001_1][Event:TASK_ATTEMPT_FINISHED]: vertexName=E205BB9BAA7744F9B0E4DC91DE388B4F, taskAttemptId=attempt_1463523506869_0001_1_00_000000_0, startTime=1463523508335, finishTime=1463523508725, timeTaken=390, status=SUCCEEDED, errorEnum=, diagnostics=, counters=Counters: 28, org.apache.tez.common.counters.DAGCounter, RACK_LOCAL_TASKS=1, File System Counters, FILE_BYTES_READ=110, FILE_BYTES_WRITTEN=727, FILE_READ_OPS=0, FILE_LARGE_READ_OPS=0, FILE_WRITE_OPS=0, org.apache.tez.common.counters.TaskCounter, SPILLED_RECORDS=13, GC_TIME_MILLIS=0, CPU_MILLISECONDS=0, PHYSICAL_MEMORY_BYTES=0, VIRTUAL_MEMORY_BYTES=0, COMMITTED_HEAP_BYTES=295174144, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=13, OUTPUT_BYTES=195, OUTPUT_BYTES_WITH_OVERHEAD=171, OUTPUT_BYTES_PHYSICAL=175, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=0, ADDITIONAL_SPILL_COUNT=0, cascading.flow.SliceCounters, Process_Begin_Time=1463523508675, Process_Duration=9, Process_End_Time=1463523508684, Read_Duration=1, Tuples_Read=13, Tuples_Written=13, Write_Duration=1, cascading.flow.StepCounters, Tuples_Read=13
2016-05-17 22:18:28,736 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:handle(692)) - attempt_1463523506869_0001_1_00_000000_0 TaskAttempt Transitioned from RUNNING to SUCCEEDED due to event TA_DONE
2016-05-17 22:18:28,736 INFO rm.TaskSchedulerEventHandler (TaskSchedulerEventHandler.java:handleEvent(149)) - Processing the event EventType: S_TA_ENDED
2016-05-17 22:18:28,739 INFO impl.TaskImpl (TaskImpl.java:transition(1052)) - Task succeeded with attempt attempt_1463523506869_0001_1_00_000000_0
2016-05-17 22:18:28,741 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event TASK_FINISHED to history file
2016-05-17 22:18:28,745 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1463523506869_0001_1][Event:TASK_FINISHED]: vertexName=E205BB9BAA7744F9B0E4DC91DE388B4F, taskId=task_1463523506869_0001_1_00_000000, startTime=1463523508335, finishTime=1463523508740, timeTaken=405, status=SUCCEEDED, successfulAttemptID=attempt_1463523506869_0001_1_00_000000_0, diagnostics=, counters=Counters: 28, org.apache.tez.common.counters.DAGCounter, RACK_LOCAL_TASKS=1, File System Counters, FILE_BYTES_READ=110, FILE_BYTES_WRITTEN=727, FILE_READ_OPS=0, FILE_LARGE_READ_OPS=0, FILE_WRITE_OPS=0, org.apache.tez.common.counters.TaskCounter, SPILLED_RECORDS=13, GC_TIME_MILLIS=0, CPU_MILLISECONDS=0, PHYSICAL_MEMORY_BYTES=0, VIRTUAL_MEMORY_BYTES=0, COMMITTED_HEAP_BYTES=295174144, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=13, OUTPUT_BYTES=195, OUTPUT_BYTES_WITH_OVERHEAD=171, OUTPUT_BYTES_PHYSICAL=175, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=0, ADDITIONAL_SPILL_COUNT=0, cascading.flow.SliceCounters, Process_Begin_Time=1463523508675, Process_Duration=9, Process_End_Time=1463523508684, Read_Duration=1, Tuples_Read=13, Tuples_Written=13, Write_Duration=1, cascading.flow.StepCounters, Tuples_Read=13
2016-05-17 22:18:28,746 INFO impl.TaskImpl (TaskImpl.java:handle(862)) - task_1463523506869_0001_1_00_000000 Task Transitioned from RUNNING to SUCCEEDED due to event T_ATTEMPT_SUCCEEDED
2016-05-17 22:18:28,748 INFO container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1463523506869_0001_00_000001 transitioned from RUNNING to IDLE via event C_TA_SUCCEEDED
2016-05-17 22:18:28,750 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event CONTAINER_STOPPED to history file
2016-05-17 22:18:28,750 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1463523506869_0001_1][Event:CONTAINER_STOPPED]: containerId=container_1463523506869_0001_00_000001, stoppedTime=1463523508750, exitStatus=0
2016-05-17 22:18:28,752 INFO container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1463523506869_0001_00_000001 transitioned from IDLE to STOP_REQUESTED via event C_STOP_REQUEST
2016-05-17 22:18:28,753 INFO impl.VertexImpl (VertexImpl.java:transition(3536)) - Num completed Tasks for vertex_1463523506869_0001_1_00 [E205BB9BAA7744F9B0E4DC91DE388B4F] : 1
2016-05-17 22:18:28,754 INFO impl.VertexImpl (VertexImpl.java:checkVertexForCompletion(1674)) - Vertex succeeded: vertex_1463523506869_0001_1_00 [E205BB9BAA7744F9B0E4DC91DE388B4F]
2016-05-17 22:18:28,764 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event VERTEX_FINISHED to history file
2016-05-17 22:18:28,769 INFO app.TaskAttemptListenerImpTezDag (TaskAttemptListenerImpTezDag.java:getTask(213)) - Container with id: container_1463523506869_0001_00_000001 is valid, but no longer registered, and will be killed
2016-05-17 22:18:28,769 INFO task.ContainerReporter (ContainerReporter.java:call(65)) - Got TaskUpdate for containerId= container_1463523506869_0001_00_000001: 38 ms after starting to poll. TaskInfo: shouldDie: true
2016-05-17 22:18:28,769 INFO task.TezChild (TezChild.java:run(207)) - ContainerTask returned shouldDie=true for container container_1463523506869_0001_00_000001, Exiting
2016-05-17 22:18:28,770 INFO task.TezChild (TezChild.java:shutdown(343)) - Shutdown invoked for container container_1463523506869_0001_00_000001
2016-05-17 22:18:28,770 INFO task.TezChild (TezChild.java:shutdown(345)) - Shutting down container container_1463523506869_0001_00_000001
2016-05-17 22:18:28,770 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1463523506869_0001_1][Event:VERTEX_FINISHED]: vertexName=E205BB9BAA7744F9B0E4DC91DE388B4F, vertexId=vertex_1463523506869_0001_1_00, initRequestedTime=1463523507999, initedTime=1463523508148, startRequestedTime=1463523508152, startedTime=1463523508153, finishTime=1463523508754, timeTaken=601, status=SUCCEEDED, diagnostics=, counters=Counters: 28, org.apache.tez.common.counters.DAGCounter, RACK_LOCAL_TASKS=1, File System Counters, FILE_BYTES_READ=110, FILE_BYTES_WRITTEN=727, FILE_READ_OPS=0, FILE_LARGE_READ_OPS=0, FILE_WRITE_OPS=0, org.apache.tez.common.counters.TaskCounter, SPILLED_RECORDS=13, GC_TIME_MILLIS=0, CPU_MILLISECONDS=0, PHYSICAL_MEMORY_BYTES=0, VIRTUAL_MEMORY_BYTES=0, COMMITTED_HEAP_BYTES=295174144, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=13, OUTPUT_BYTES=195, OUTPUT_BYTES_WITH_OVERHEAD=171, OUTPUT_BYTES_PHYSICAL=175, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=0, ADDITIONAL_SPILL_COUNT=0, cascading.flow.SliceCounters, Process_Begin_Time=1463523508675, Process_Duration=9, Process_End_Time=1463523508684, Read_Duration=1, Tuples_Read=13, Tuples_Written=13, Write_Duration=1, cascading.flow.StepCounters, Tuples_Read=13, vertexStats=firstTaskStartTime=1463523508335, firstTasksToStart=[ task_1463523506869_0001_1_00_000000 ], lastTaskFinishTime=1463523508725, lastTasksToFinish=[ task_1463523506869_0001_1_00_000000 ], minTaskDuration=390, maxTaskDuration=390, avgTaskDuration=390.0, numSuccessfulTasks=1, shortestDurationTasks=[ task_1463523506869_0001_1_00_000000 ], longestDurationTasks=[ task_1463523506869_0001_1_00_000000 ], vertexTaskStats={numFailedTasks=0, numSucceededTasks=1, numKilledTaskAttempts=0, numKilledTasks=0, numFailedTaskAttempts=0, numCompletedTasks=1}
2016-05-17 22:18:28,773 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(270)) - ContainerExecutionResult for: container_1463523506869_0001_00_000001 = ContainerExecutionResult{exitStatus=SUCCESS, throwable=null, errorMessage='Asked to die by the AM'}
2016-05-17 22:18:28,773 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(274)) - Container: container_1463523506869_0001_00_000001 completed successfully
2016-05-17 22:18:28,773 INFO impl.VertexImpl (VertexImpl.java:handle(1564)) - vertex_1463523506869_0001_1_00 [E205BB9BAA7744F9B0E4DC91DE388B4F] transitioned from RUNNING to SUCCEEDED due to event V_TASK_COMPLETED
2016-05-17 22:18:28,774 INFO impl.VertexImpl (VertexImpl.java:transition(3477)) - Source task attempt completed for vertex: vertex_1463523506869_0001_1_01 [E7D2E5C67A994389B15E179A53A02836] attempt: attempt_1463523506869_0001_1_00_000000_0 with state: SUCCEEDED vertexState: RUNNING
2016-05-17 22:18:28,774 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:stop(242)) - Ignoring stop request for containerId: container_1463523506869_0001_00_000001
2016-05-17 22:18:28,774 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:schedulePendingTasks(608)) - All source tasks assigned. Ramping up 1 remaining tasks for vertex: E7D2E5C67A994389B15E179A53A02836
2016-05-17 22:18:28,776 INFO impl.VertexImpl (VertexImpl.java:scheduleTasks(1250)) - Routing pending task events for vertex: vertex_1463523506869_0001_1_01 [E7D2E5C67A994389B15E179A53A02836]
2016-05-17 22:18:28,777 INFO impl.DAGImpl (DAGImpl.java:transition(1747)) - Vertex vertex_1463523506869_0001_1_00 [E205BB9BAA7744F9B0E4DC91DE388B4F] completed., numCompletedVertices=1, numSuccessfulVertices=1, numFailedVertices=0, numKilledVertices=0, numVertices=2
2016-05-17 22:18:28,778 INFO container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1463523506869_0001_00_000001 transitioned from STOP_REQUESTED to STOPPING via event C_NM_STOP_SENT
2016-05-17 22:18:28,778 INFO container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1463523506869_0001_00_000001 transitioned from STOPPING to COMPLETED via event C_COMPLETED
2016-05-17 22:18:28,779 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event TASK_STARTED to history file
2016-05-17 22:18:28,779 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1463523506869_0001_1][Event:TASK_STARTED]: vertexName=E7D2E5C67A994389B15E179A53A02836, taskId=task_1463523506869_0001_1_01_000000, scheduledTime=1463523508778, launchTime=1463523508778
2016-05-17 22:18:28,779 INFO impl.TaskImpl (TaskImpl.java:handle(862)) - task_1463523506869_0001_1_01_000000 Task Transitioned from NEW to SCHEDULED due to event T_SCHEDULE
2016-05-17 22:18:28,780 INFO impl.DAGSchedulerNaturalOrderControlled (DAGSchedulerNaturalOrderControlled.java:scheduleTask(111)) - Scheduled vertex: vertex_1463523506869_0001_1_01 [E7D2E5C67A994389B15E179A53A02836]
2016-05-17 22:18:28,780 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:transition(1014)) - remoteTaskSpec:DAGName : [A69585C002A74BF4ABAEEC706FE1A188/A9088A6DF9614808BCA4DF4454CB724E] test/(1/1) ...stIdentityBuffer/identity, VertexName: E7D2E5C67A994389B15E179A53A02836, VertexParallelism: 1, TaskAttemptID:attempt_1463523506869_0001_1_01_000000_0, processorName=cascading.flow.tez.FlowProcessor, inputSpecListSize=1, outputSpecListSize=1, inputSpecList=[{{ sourceVertexName=E205BB9BAA7744F9B0E4DC91DE388B4F, physicalEdgeCount=1, inputClassName=org.apache.tez.runtime.library.input.OrderedGroupedKVInput }}, ], outputSpecList=[{{ destinationVertexName=D856B73605D74B1A813E6470D8E6523D, physicalEdgeCount=0, outputClassName=org.apache.tez.mapreduce.output.MROutput }}, ]
2016-05-17 22:18:28,780 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:handle(692)) - attempt_1463523506869_0001_1_01_000000_0 TaskAttempt Transitioned from NEW to START_WAIT due to event TA_SCHEDULE
2016-05-17 22:18:28,780 INFO rm.TaskSchedulerEventHandler (TaskSchedulerEventHandler.java:handleEvent(149)) - Processing the event EventType: S_TA_LAUNCH_REQUEST
2016-05-17 22:18:28,782 INFO container.AMContainerImpl (AMContainerImpl.java:transition(521)) - Sending Launch Request for Container with id: container_1463523506869_0001_00_000002
2016-05-17 22:18:28,782 INFO container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1463523506869_0001_00_000002 transitioned from ALLOCATED to LAUNCHING via event C_LAUNCH_REQUEST
2016-05-17 22:18:28,785 INFO task.TezChild (TezChild.java:newTezChild(432)) - PID, containerIdentifier: null, container_1463523506869_0001_00_000002
2016-05-17 22:18:28,785 WARN impl.MetricsConfig (MetricsConfig.java:loadFirst(124)) - Cannot locate configuration: tried hadoop-metrics2-teztask.properties,hadoop-metrics2.properties
2016-05-17 22:18:28,787 INFO container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1463523506869_0001_00_000002 transitioned from LAUNCHING to IDLE via event C_LAUNCHED
2016-05-17 22:18:28,788 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event CONTAINER_LAUNCHED to history file
2016-05-17 22:18:28,788 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1463523506869_0001_1][Event:CONTAINER_LAUNCHED]: containerId=container_1463523506869_0001_00_000002, launchTime=1463523508787
2016-05-17 22:18:28,788 INFO task.ContainerReporter (ContainerReporter.java:call(55)) - Attempting to fetch new task for container container_1463523506869_0001_00_000002
2016-05-17 22:18:28,789 INFO container.AMContainerImpl (AMContainerImpl.java:transition(827)) - Assigned taskAttempt + [attempt_1463523506869_0001_1_01_000000_0] to container: [container_1463523506869_0001_00_000002]
2016-05-17 22:18:28,789 INFO container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1463523506869_0001_00_000002 transitioned from IDLE to RUNNING via event C_PULL_TA
2016-05-17 22:18:28,791 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:transition(1146)) - TaskAttempt: [attempt_1463523506869_0001_1_01_000000_0] started. Is using containerId: [container_1463523506869_0001_00_000002] on NM: [127.0.0.1:0]
2016-05-17 22:18:28,792 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event TASK_ATTEMPT_STARTED to history file
2016-05-17 22:18:28,792 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1463523506869_0001_1][Event:TASK_ATTEMPT_STARTED]: vertexName=E7D2E5C67A994389B15E179A53A02836, taskAttemptId=attempt_1463523506869_0001_1_01_000000_0, startTime=1463523508791, containerId=container_1463523506869_0001_00_000002, nodeId=127.0.0.1:0, inProgressLogs=127.0.0.1:0/node/containerlogs/container_1463523506869_0001_00_000002/teamcity, completedLogs=
2016-05-17 22:18:28,792 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:handle(692)) - attempt_1463523506869_0001_1_01_000000_0 TaskAttempt Transitioned from START_WAIT to RUNNING due to event TA_STARTED_REMOTELY
2016-05-17 22:18:28,792 INFO impl.TaskImpl (TaskImpl.java:handle(862)) - task_1463523506869_0001_1_01_000000 Task Transitioned from SCHEDULED to RUNNING due to event T_ATTEMPT_LAUNCHED
2016-05-17 22:18:28,792 INFO app.TaskAttemptListenerImpTezDag (TaskAttemptListenerImpTezDag.java:getTask(240)) - Container with id: container_1463523506869_0001_00_000002 given task: attempt_1463523506869_0001_1_01_000000_0
2016-05-17 22:18:28,793 INFO task.ContainerReporter (ContainerReporter.java:call(65)) - Got TaskUpdate for containerId= container_1463523506869_0001_00_000002: 0 ms after starting to poll. TaskInfo: shouldDie: false, currentTaskAttemptId: attempt_1463523506869_0001_1_01_000000_0
2016-05-17 22:18:28 Starting to run new task attempt: attempt_1463523506869_0001_1_01_000000_0
2016-05-17 22:18:28,793 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(164)) - Redirecting log file based on addend: attempt_1463523506869_0001_1_01_000000_0
2016-05-17 22:18:28,794 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(179)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-17 22:18:28,794 INFO task.TezChild (TezChild.java:handleNewTaskCredentials(273)) - Refreshing UGI since Credentials have changed
2016-05-17 22:18:28,794 INFO task.TezChild (TezChild.java:handleNewTaskCredentials(276)) - Credentials : #Tokens=1, #SecretKeys=0
2016-05-17 22:18:28,794 INFO task.TezChild (TezChild.java:handleNewTaskLocalResources(302)) - Localizing additional local resources for Task : {}
2016-05-17 22:18:28,795 INFO task.TezChild (TezChild.java:handleNewTaskLocalResources(312)) - Done localizing additional resources
2016-05-17 22:18:28,795 INFO metrics.TaskCounterUpdater (TaskCounterUpdater.java:initResourceCalculatorPlugin(148)) - Using ResourceCalculatorProcessTree : [ ]
2016-05-17 22:18:28,795 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:<init>(149)) - Initializing LogicalIOProcessorRuntimeTask with TaskSpec: DAGName : [A69585C002A74BF4ABAEEC706FE1A188/A9088A6DF9614808BCA4DF4454CB724E] test/(1/1) ...stIdentityBuffer/identity, VertexName: E7D2E5C67A994389B15E179A53A02836, VertexParallelism: 1, TaskAttemptID:attempt_1463523506869_0001_1_01_000000_0, processorName=cascading.flow.tez.FlowProcessor, inputSpecListSize=1, outputSpecListSize=1, inputSpecList=[{{ sourceVertexName=E205BB9BAA7744F9B0E4DC91DE388B4F, physicalEdgeCount=1, inputClassName=org.apache.tez.runtime.library.input.OrderedGroupedKVInput }}, ], outputSpecList=[{{ destinationVertexName=D856B73605D74B1A813E6470D8E6523D, physicalEdgeCount=0, outputClassName=org.apache.tez.mapreduce.output.MROutput }}, ]
2016-05-17 22:18:28,796 INFO resources.MemoryDistributor (MemoryDistributor.java:<init>(85)) - InitialMemoryDistributor (isEnabled=true) invoked with: numInputs=1, numOutputs=1, JVM.maxFree=704643072
2016-05-17 22:18:28,796 INFO task.TezTaskRunner (TezTaskRunner.java:run(172)) - Initializing task, taskAttemptId=attempt_1463523506869_0001_1_01_000000_0
2016-05-17 22:18:28,800 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(191)) - Initializing LogicalProcessorIORuntimeTask
2016-05-17 22:18:28,800 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(195)) - Creating processor, processorClassName=cascading.flow.tez.FlowProcessor
2016-05-17 22:18:28,803 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initializeLogicalIOProcessor(488)) - Initializing processor, processorClassName=cascading.flow.tez.FlowProcessor
2016-05-17 22:18:28,803 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(438)) - Initializing Output using OutputSpec: { destinationVertexName=D856B73605D74B1A813E6470D8E6523D, physicalEdgeCount=0, outputClassName=org.apache.tez.mapreduce.output.MROutput }
2016-05-17 22:18:28,804 INFO impl.TaskImpl (TaskImpl.java:getTaskAttemptTezEvents(481)) - TaskAttempt:attempt_1463523506869_0001_1_01_000000_0 sent events: (0-1)
2016-05-17 22:18:28,805 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:createOutput(565)) - Creating Output
2016-05-17 22:18:28,805 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(446)) - Initializing Output with dest edge: D856B73605D74B1A813E6470D8E6523D
2016-05-17 22:18:28,805 INFO output.MROutput (MROutput.java:initialize(347)) - Initializing Simple Output
2016-05-17 22:18:28,806 INFO resources.MemoryDistributor$RequestorInfo (MemoryDistributor.java:<init>(233)) - Received request: 0, type: OUTPUT, componentVertexName: D856B73605D74B1A813E6470D8E6523D
2016-05-17 22:18:28,807 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(389)) - Initializing Input using InputSpec: { sourceVertexName=E205BB9BAA7744F9B0E4DC91DE388B4F, physicalEdgeCount=1, inputClassName=org.apache.tez.runtime.library.input.OrderedGroupedKVInput }
2016-05-17 22:18:28,809 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:createInput(537)) - Creating Input
2016-05-17 22:18:28,813 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(397)) - Initializing Input with src edge: E205BB9BAA7744F9B0E4DC91DE388B4F
2016-05-17 22:18:28,811 INFO tez.FlowProcessor (FlowProcessor.java:initialize(88)) - cascading version: 3.1.0
2016-05-17 22:18:28,833 INFO tez.FlowProcessor (FlowProcessor.java:initialize(94)) - flow node id: E7D2E5C67A994389B15E179A53A02836, ordinal: 1
2016-05-17 22:18:28,833 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: E7D2E5C67A994389B15E179A53A02836, mem on start (mb), free: 133, total: 281, max: 672
2016-05-17 22:18:28,834 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initializeLogicalIOProcessor(491)) - Initialized processor, processorClassName=cascading.flow.tez.FlowProcessor
2016-05-17 22:18:28,834 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(220)) - Waiting for 2 initializers to finish
2016-05-17 22:18:28,836 INFO orderedgrouped.MergeManager (MergeManager.java:getInitialMemoryRequirement(310)) - Initial Shuffle Memory Required: 634178752, based on INPUT_BUFFER_factor: 0.9
2016-05-17 22:18:28,837 INFO orderedgrouped.MergeManager (MergeManager.java:getInitialMemoryRequirement(320)) - Initial Memory required for final merged output: 0, using factor: 0.0
2016-05-17 22:18:28,837 INFO resources.MemoryDistributor$RequestorInfo (MemoryDistributor.java:<init>(233)) - Received request: 634178752, type: INPUT, componentVertexName: E205BB9BAA7744F9B0E4DC91DE388B4F
2016-05-17 22:18:28,839 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(403)) - Initialized Input with src edge: E205BB9BAA7744F9B0E4DC91DE388B4F
2016-05-17 22:18:28,839 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(220)) - Waiting for 1 initializers to finish
2016-05-17 22:18:28,840 INFO output.MROutput (MROutput.java:initialize(418)) - Initialized Simple Output, using_new_api: false
2016-05-17 22:18:28,841 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(452)) - Initialized Output with dest edge: D856B73605D74B1A813E6470D8E6523D
2016-05-17 22:18:28,842 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(233)) - All initializers finished
2016-05-17 22:18:28,843 INFO resources.MemoryDistributor (MemoryDistributor.java:makeInitialAllocations(124)) - Using Allocator class: org.apache.tez.runtime.library.resources.WeightedScalingMemoryDistributor
2016-05-17 22:18:28,844 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:getRequestTypeForClass(197)) - Falling back to RequestType.OTHER for class: org.apache.tez.mapreduce.output.MROutput
2016-05-17 22:18:28,845 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:initialProcessMemoryRequestContext(171)) - ScaleFactor: 1, for type: OTHER
2016-05-17 22:18:28,846 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:initialProcessMemoryRequestContext(171)) - ScaleFactor: 12, for type: SORTED_MERGED_INPUT
2016-05-17 22:18:28,846 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:computeReservedFraction(262)) - InitialReservationFraction=0.3, AdditionalReservationFractionForIOs=0.03, finalReserveFractionUsed=0.32999999999999996
2016-05-17 22:18:28,847 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:assignMemory(126)) - Scaling Requests. NumRequests: 2, numScaledRequests: 13, TotalRequested: 634178752, TotalRequestedScaled: 5.853957710769231E8, TotalJVMHeap: 704643072, TotalAvailable: 472110858, TotalRequested/TotalJVMHeap:0.90
2016-05-17 22:18:28,847 INFO resources.MemoryDistributor (MemoryDistributor.java:makeInitialAllocations(138)) - Informing: OUTPUT, D856B73605D74B1A813E6470D8E6523D, org.apache.tez.mapreduce.output.MROutput: requested=0, allocated=0
2016-05-17 22:18:28,847 INFO resources.MemoryDistributor (MemoryDistributor.java:makeInitialAllocations(138)) - Informing: INPUT, E205BB9BAA7744F9B0E4DC91DE388B4F, org.apache.tez.runtime.library.input.OrderedGroupedKVInput: requested=634178752, allocated=472110858
2016-05-17 22:18:28,848 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(255)) - Starting Inputs/Outputs
2016-05-17 22:18:28,848 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(268)) - Input: E205BB9BAA7744F9B0E4DC91DE388B4F being auto started by the framework. Subsequent instances will not be auto-started
2016-05-17 22:18:28,848 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(419)) - Starting Input with src edge: E205BB9BAA7744F9B0E4DC91DE388B4F
2016-05-17 22:18:28,848 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(289)) - Num IOs determined for AutoStart: 1
2016-05-17 22:18:28,850 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(291)) - Waiting for 1 IOs to start
2016-05-17 22:18:28,856 INFO common.TezRuntimeUtils (TezRuntimeUtils.java:instantiateCombiner(65)) - No combiner specified via tez.runtime.combiner.class. Combiner will not be used
2016-05-17 22:18:28,857 INFO orderedgrouped.Shuffle (Shuffle.java:<init>(182)) - Shuffle assigned with 1 inputs, codec: NoneifileReadAhead: true
2016-05-17 22:18:28,860 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:<init>(158)) - ShuffleScheduler running for sourceVertex: E205BB9BAA7744F9B0E4DC91DE388B4F with configuration: maxFetchFailuresBeforeReporting=5, reportReadErrorImmediately=true, maxFailedUniqueFetches=1, abortFailureLimit=30, maxMapRuntime=0, maxTaskOutputAtOnce=20
2016-05-17 22:18:28,862 INFO orderedgrouped.MergeManager (MergeManager.java:<init>(221)) - InitialRequest: ShuffleMem=634178752, postMergeMem=0, RuntimeTotalAvailable=472110858. Updated to: ShuffleMem=472110858, postMergeMem: 0
2016-05-17 22:18:28,863 INFO orderedgrouped.MergeManager (MergeManager.java:<init>(252)) - MergerManager: memoryLimit=472110858, maxSingleShuffleLimit=118027712, mergeThreshold=424899744, ioSortFactor=100, memToMemMergeOutputsThreshold=100
2016-05-17 22:18:28,866 INFO orderedgrouped.Shuffle (Shuffle.java:<init>(233)) - Num fetchers being started: 1
2016-05-17 22:18:28,871 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(421)) - Started Input with src edge: E205BB9BAA7744F9B0E4DC91DE388B4F
2016-05-17 22:18:28,871 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(304)) - AutoStartComplete
2016-05-17 22:18:28,875 INFO task.TezTaskRunner (TezTaskRunner.java:run(175)) - Running task, taskAttemptId=attempt_1463523506869_0001_1_01_000000_0
2016-05-17 22:18:28,877 INFO orderedgrouped.ShuffleInputEventHandlerOrderedGrouped (ShuffleInputEventHandlerOrderedGrouped.java:processDataMovementEvent(84)) - DME srcIdx: 0, targetIdx: 0, attemptNum: 0, payload: [host: ip-10-123-148-208, port: 0, pathComponent: attempt_1463523506869_0001_1_00_000000_0_10003, runDuration: 310793, ]
2016-05-17 22:18:28,885 INFO graph.Hadoop2TezStreamGraph (Hadoop2TezStreamGraph.java:buildGraph(151)) - using streamed source: GroupBy(test)[by:[{1}:'num']]
2016-05-17 22:18:28,886 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:logIndividualFetchComplete(250)) - Completed fetch for attempt: InputAttemptIdentifier [inputIdentifier=InputIdentifier [inputIndex=0], attemptNumber=0, pathComponent=attempt_1463523506869_0001_1_00_000000_0_10003] to DISK_DIRECT, CompressedSize=175, DecompressedSize=171,EndTime=1463523508886, TimeTaken=3, Rate=0.06 MB/s
2016-05-17 22:18:28,887 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:copySucceeded(213)) - All inputs fetched for input vertex : E205BB9BAA7744F9B0E4DC91DE388B4F
2016-05-17 22:18:28,888 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:logProgress(263)) - copy(1 of 1. Transfer rate (CumulativeDataFetched/TimeSinceInputStarted)) 0.00 MB/s)
2016-05-17 22:18:28,889 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:freeHost(529)) - ip-10-123-148-208:0 freed by fetcher [E205BB9BAA7744F9B0E4DC91DE388B4F] #1 in 6ms
2016-05-17 22:18:28,889 INFO orderedgrouped.Shuffle (Shuffle.java:cleanupFetchers(394)) - Shutdown..fetcher [E205BB9BAA7744F9B0E4DC91DE388B4F] #1, status:false, isInterrupted:false
2016-05-17 22:18:28,892 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(854)) - finalMerge called with 0 in-memory map-outputs and 1 on-disk map-outputs
2016-05-17 22:18:28,896 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(951)) - Merging 1 files, 175 bytes from disk
2016-05-17 22:18:28,897 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(966)) - Merging 0 segments, 0 bytes from memory into reduce
2016-05-17 22:18:28,900 INFO tez.FlowProcessor (FlowProcessor.java:run(121)) - sourcing from: GroupBy(test)[by:[{1}:'num']] streamed: true, id: 76E96E85058B485EB8C48862CCECFE63
2016-05-17 22:18:28,901 INFO tez.FlowProcessor (FlowProcessor.java:run(124)) - sinking to: Hfs["TextLine[['line']->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.5.x-tez-0.6.x/build/test/output/hadoop2-tez/bufferpipesplatform/testIdentityBuffer/identity"], id: D856B73605D74B1A813E6470D8E6523D
2016-05-17 22:18:28,902 INFO element.TezSinkStage (TezSinkStage.java:prepare(57)) - calling MROutput#start() on: Hfs["TextLine[['line']->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.5.x-tez-0.6.x/build/test/output/hadoop2-tez/bufferpipesplatform/testIdentityBuffer/identity"]
2016-05-17 22:18:28,903 INFO element.TezGroupGate (TezGroupGate.java:prepare(90)) - calling OrderedGroupedKVInput#start() on: GroupBy(test)[by:[{1}:'num']] 76E96E85058B485EB8C48862CCECFE63, for 1 inputs
2016-05-17 22:18:28,905 INFO impl.TezMerger (TezMerger.java:merge(560)) - Merging 1 sorted segments
2016-05-17 22:18:28,908 INFO impl.TezMerger (TezMerger.java:merge(665)) - Down to the last merge-pass, with 1 segments left of total size: 158 bytes
2016-05-17 22:18:28,908 INFO orderedgrouped.Shuffle (Shuffle.java:call(381)) - merge complete for input vertex : E205BB9BAA7744F9B0E4DC91DE388B4F
2016-05-17 22:18:28,909 INFO orderedgrouped.Shuffle (Shuffle.java:onSuccess(488)) - Shuffle Runner thread complete
2016-05-17 22:18:28,909 INFO tez.FlowProcessor (FlowProcessor.java:waitForInputsReady(202)) - flow node id: E7D2E5C67A994389B15E179A53A02836, all 1 inputs ready in: 00:00:00.005
2016-05-17 22:18:28,915 INFO common.ValuesIterator (ValuesIterator.java:<init>(83)) - keyDeserializer=cascading.tuple.hadoop.io.KeyTupleDeserializer@7b390e5e; valueDeserializer=cascading.tuple.hadoop.io.ValueTupleDeserializer@5ede7c85
2016-05-17 22:18:28,925 INFO output.MROutput (MROutput.java:flush(534)) - Flushing Simple Output
2016-05-17 22:18:28,926 INFO output.MROutput (MROutput.java:flush(544)) - Flushed Simple Output
2016-05-17 22:18:28,927 INFO app.TaskAttemptListenerImpTezDag (TaskAttemptListenerImpTezDag.java:canCommit(353)) - Commit go/no-go request from attempt_1463523506869_0001_1_01_000000_0
2016-05-17 22:18:28,927 INFO impl.TaskImpl (TaskImpl.java:canCommit(750)) - attempt_1463523506869_0001_1_01_000000_0 given a go for committing the task output.
2016-05-17 22:18:28,929 INFO task.TezTaskRunner (TezTaskRunner.java:run(177)) - Closing task, taskAttemptId=attempt_1463523506869_0001_1_01_000000_0
2016-05-17 22:18:28,929 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: E7D2E5C67A994389B15E179A53A02836, mem on close (mb), free: 125, total: 281, max: 672
2016-05-17 22:18:28,930 INFO orderedgrouped.Shuffle (Shuffle.java:shutdown(320)) - Shutting down Shuffle for source: E205BB9BAA7744F9B0E4DC91DE388B4F
2016-05-17 22:18:28,930 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(854)) - finalMerge called with 0 in-memory map-outputs and 0 on-disk map-outputs
2016-05-17 22:18:28,932 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(951)) - Merging 0 files, 0 bytes from disk
2016-05-17 22:18:28,932 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(966)) - Merging 0 segments, 0 bytes from memory into reduce
2016-05-17 22:18:28,932 INFO impl.TezMerger (TezMerger.java:merge(560)) - Merging 0 sorted segments
2016-05-17 22:18:28,933 INFO impl.TezMerger (TezMerger.java:merge(562)) - Nothing to merge. Returning an empty iterator
2016-05-17 22:18:28 Completed running task attempt: attempt_1463523506869_0001_1_01_000000_0
2016-05-17 22:18:28,935 INFO task.TezTaskRunner (TezTaskRunner.java:run(181)) - Task completed, taskAttemptId=attempt_1463523506869_0001_1_01_000000_0, fatalErrorOccurred=false
2016-05-17 22:18:28,938 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event TASK_ATTEMPT_FINISHED to history file
2016-05-17 22:18:28,940 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:cleanup(712)) - Final Counters : Counters: 46 [[File System Counters FILE_BYTES_READ=207, FILE_BYTES_WRITTEN=64, FILE_READ_OPS=0, FILE_LARGE_READ_OPS=0, FILE_WRITE_OPS=0][org.apache.tez.common.counters.TaskCounter REDUCE_INPUT_GROUPS=4, REDUCE_INPUT_RECORDS=13, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=13, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, GC_TIME_MILLIS=0, CPU_MILLISECONDS=0, PHYSICAL_MEMORY_BYTES=0, VIRTUAL_MEMORY_BYTES=0, COMMITTED_HEAP_BYTES=295174144, OUTPUT_RECORDS=13, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=175, SHUFFLE_BYTES=175, SHUFFLE_BYTES_DECOMPRESSED=171, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=175, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=32, MERGE_PHASE_TIME=51, FIRST_EVENT_RECEIVED=26, LAST_EVENT_RECEIVED=26][Shuffle Errors BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0][cascading.flow.SliceCounters Process_Begin_Time=1463523508911, Process_Duration=18, Process_End_Time=1463523508929, Read_Duration=3, Tuples_Read=13, Tuples_Written=13, Write_Duration=1][cascading.flow.StepCounters Tuples_Written=13]]
2016-05-17 22:18:28,943 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1463523506869_0001_1][Event:TASK_ATTEMPT_FINISHED]: vertexName=E7D2E5C67A994389B15E179A53A02836, taskAttemptId=attempt_1463523506869_0001_1_01_000000_0, startTime=1463523508791, finishTime=1463523508938, timeTaken=147, status=SUCCEEDED, errorEnum=, diagnostics=, counters=Counters: 46, File System Counters, FILE_BYTES_READ=207, FILE_BYTES_WRITTEN=64, FILE_READ_OPS=0, FILE_LARGE_READ_OPS=0, FILE_WRITE_OPS=0, org.apache.tez.common.counters.TaskCounter, REDUCE_INPUT_GROUPS=4, REDUCE_INPUT_RECORDS=13, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=13, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, GC_TIME_MILLIS=0, CPU_MILLISECONDS=0, PHYSICAL_MEMORY_BYTES=0, VIRTUAL_MEMORY_BYTES=0, COMMITTED_HEAP_BYTES=295174144, OUTPUT_RECORDS=13, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=175, SHUFFLE_BYTES=175, SHUFFLE_BYTES_DECOMPRESSED=171, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=175, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=32, MERGE_PHASE_TIME=51, FIRST_EVENT_RECEIVED=26, LAST_EVENT_RECEIVED=26, Shuffle Errors, BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0, cascading.flow.SliceCounters, Process_Begin_Time=1463523508911, Process_Duration=18, Process_End_Time=1463523508929, Read_Duration=3, Tuples_Read=13, Tuples_Written=13, Write_Duration=1, cascading.flow.StepCounters, Tuples_Written=13
2016-05-17 22:18:28,944 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:handle(692)) - attempt_1463523506869_0001_1_01_000000_0 TaskAttempt Transitioned from RUNNING to SUCCEEDED due to event TA_DONE
2016-05-17 22:18:28,944 INFO rm.TaskSchedulerEventHandler (TaskSchedulerEventHandler.java:handleEvent(149)) - Processing the event EventType: S_TA_ENDED
2016-05-17 22:18:28,944 INFO impl.TaskImpl (TaskImpl.java:transition(1052)) - Task succeeded with attempt attempt_1463523506869_0001_1_01_000000_0
2016-05-17 22:18:28,945 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event TASK_FINISHED to history file
2016-05-17 22:18:28,947 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(164)) - Redirecting log file based on addend:
2016-05-17 22:18:28,947 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(179)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-17 22:18:28,948 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1463523506869_0001_1][Event:TASK_FINISHED]: vertexName=E7D2E5C67A994389B15E179A53A02836, taskId=task_1463523506869_0001_1_01_000000, startTime=1463523508791, finishTime=1463523508944, timeTaken=153, status=SUCCEEDED, successfulAttemptID=attempt_1463523506869_0001_1_01_000000_0, diagnostics=, counters=Counters: 46, File System Counters, FILE_BYTES_READ=207, FILE_BYTES_WRITTEN=64, FILE_READ_OPS=0, FILE_LARGE_READ_OPS=0, FILE_WRITE_OPS=0, org.apache.tez.common.counters.TaskCounter, REDUCE_INPUT_GROUPS=4, REDUCE_INPUT_RECORDS=13, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=13, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, GC_TIME_MILLIS=0, CPU_MILLISECONDS=0, PHYSICAL_MEMORY_BYTES=0, VIRTUAL_MEMORY_BYTES=0, COMMITTED_HEAP_BYTES=295174144, OUTPUT_RECORDS=13, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=175, SHUFFLE_BYTES=175, SHUFFLE_BYTES_DECOMPRESSED=171, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=175, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=32, MERGE_PHASE_TIME=51, FIRST_EVENT_RECEIVED=26, LAST_EVENT_RECEIVED=26, Shuffle Errors, BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0, cascading.flow.SliceCounters, Process_Begin_Time=1463523508911, Process_Duration=18, Process_End_Time=1463523508929, Read_Duration=3, Tuples_Read=13, Tuples_Written=13, Write_Duration=1, cascading.flow.StepCounters, Tuples_Written=13
2016-05-17 22:18:28,948 INFO impl.TaskImpl (TaskImpl.java:handle(862)) - task_1463523506869_0001_1_01_000000 Task Transitioned from RUNNING to SUCCEEDED due to event T_ATTEMPT_SUCCEEDED
2016-05-17 22:18:28,948 INFO impl.VertexImpl (VertexImpl.java:transition(3536)) - Num completed Tasks for vertex_1463523506869_0001_1_01 [E7D2E5C67A994389B15E179A53A02836] : 1
2016-05-17 22:18:28,948 INFO impl.VertexImpl (VertexImpl.java:checkVertexForCompletion(1674)) - Vertex succeeded: vertex_1463523506869_0001_1_01 [E7D2E5C67A994389B15E179A53A02836]
2016-05-17 22:18:28,948 INFO task.ContainerReporter (ContainerReporter.java:call(55)) - Attempting to fetch new task for container container_1463523506869_0001_00_000002
2016-05-17 22:18:28,950 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event VERTEX_FINISHED to history file
2016-05-17 22:18:28,953 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1463523506869_0001_1][Event:VERTEX_FINISHED]: vertexName=E7D2E5C67A994389B15E179A53A02836, vertexId=vertex_1463523506869_0001_1_01, initRequestedTime=1463523508046, initedTime=1463523508127, startRequestedTime=1463523508207, startedTime=1463523508207, finishTime=1463523508949, timeTaken=742, status=SUCCEEDED, diagnostics=, counters=Counters: 46, File System Counters, FILE_BYTES_READ=207, FILE_BYTES_WRITTEN=64, FILE_READ_OPS=0, FILE_LARGE_READ_OPS=0, FILE_WRITE_OPS=0, org.apache.tez.common.counters.TaskCounter, REDUCE_INPUT_GROUPS=4, REDUCE_INPUT_RECORDS=13, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=13, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, GC_TIME_MILLIS=0, CPU_MILLISECONDS=0, PHYSICAL_MEMORY_BYTES=0, VIRTUAL_MEMORY_BYTES=0, COMMITTED_HEAP_BYTES=295174144, OUTPUT_RECORDS=13, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=175, SHUFFLE_BYTES=175, SHUFFLE_BYTES_DECOMPRESSED=171, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=175, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=32, MERGE_PHASE_TIME=51, FIRST_EVENT_RECEIVED=26, LAST_EVENT_RECEIVED=26, Shuffle Errors, BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0, cascading.flow.SliceCounters, Process_Begin_Time=1463523508911, Process_Duration=18, Process_End_Time=1463523508929, Read_Duration=3, Tuples_Read=13, Tuples_Written=13, Write_Duration=1, cascading.flow.StepCounters, Tuples_Written=13, vertexStats=firstTaskStartTime=1463523508791, firstTasksToStart=[ task_1463523506869_0001_1_01_000000 ], lastTaskFinishTime=1463523508938, lastTasksToFinish=[ task_1463523506869_0001_1_01_000000 ], minTaskDuration=147, maxTaskDuration=147, avgTaskDuration=147.0, numSuccessfulTasks=1, shortestDurationTasks=[ task_1463523506869_0001_1_01_000000 ], longestDurationTasks=[ task_1463523506869_0001_1_01_000000 ], vertexTaskStats={numFailedTasks=0, numSucceededTasks=1, numKilledTaskAttempts=0, numKilledTasks=0, numFailedTaskAttempts=0, numCompletedTasks=1}
2016-05-17 22:18:28,954 INFO impl.VertexImpl (VertexImpl.java:handle(1564)) - vertex_1463523506869_0001_1_01 [E7D2E5C67A994389B15E179A53A02836] transitioned from RUNNING to SUCCEEDED due to event V_TASK_COMPLETED
2016-05-17 22:18:28,955 INFO container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1463523506869_0001_00_000002 transitioned from RUNNING to IDLE via event C_TA_SUCCEEDED
2016-05-17 22:18:28,956 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event CONTAINER_STOPPED to history file
2016-05-17 22:18:28,956 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1463523506869_0001_1][Event:CONTAINER_STOPPED]: containerId=container_1463523506869_0001_00_000002, stoppedTime=1463523508956, exitStatus=0
2016-05-17 22:18:28,957 INFO container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1463523506869_0001_00_000002 transitioned from IDLE to STOP_REQUESTED via event C_STOP_REQUEST
2016-05-17 22:18:28,958 INFO impl.DAGImpl (DAGImpl.java:transition(1747)) - Vertex vertex_1463523506869_0001_1_01 [E7D2E5C67A994389B15E179A53A02836] completed., numCompletedVertices=2, numSuccessfulVertices=2, numFailedVertices=0, numKilledVertices=0, numVertices=2
2016-05-17 22:18:28,958 INFO impl.DAGImpl (DAGImpl.java:commitOrAbortOutputs(831)) - Calling DAG commit/abort for dag: dag_1463523506869_0001_1
2016-05-17 22:18:28,960 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1463523506869_0001_1][Event:DAG_COMMIT_STARTED]: dagID=dag_1463523506869_0001_1
2016-05-17 22:18:28,960 INFO impl.DAGImpl (DAGImpl.java:commitOrAbortOutputs(892)) - Committing output: D856B73605D74B1A813E6470D8E6523D for vertex: vertex_1463523506869_0001_1_01 [E7D2E5C67A994389B15E179A53A02836]
2016-05-17 22:18:28,962 INFO app.TaskAttemptListenerImpTezDag (TaskAttemptListenerImpTezDag.java:getTask(213)) - Container with id: container_1463523506869_0001_00_000002 is valid, but no longer registered, and will be killed
2016-05-17 22:18:28,962 INFO task.ContainerReporter (ContainerReporter.java:call(65)) - Got TaskUpdate for containerId= container_1463523506869_0001_00_000002: 12 ms after starting to poll. TaskInfo: shouldDie: true
2016-05-17 22:18:28,962 INFO task.TezChild (TezChild.java:run(207)) - ContainerTask returned shouldDie=true for container container_1463523506869_0001_00_000002, Exiting
2016-05-17 22:18:28,962 INFO task.TezChild (TezChild.java:shutdown(343)) - Shutdown invoked for container container_1463523506869_0001_00_000002
2016-05-17 22:18:28,963 INFO task.TezChild (TezChild.java:shutdown(345)) - Shutting down container container_1463523506869_0001_00_000002
2016-05-17 22:18:28,964 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(270)) - ContainerExecutionResult for: container_1463523506869_0001_00_000002 = ContainerExecutionResult{exitStatus=SUCCESS, throwable=null, errorMessage='Asked to die by the AM'}
2016-05-17 22:18:28,965 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(274)) - Container: container_1463523506869_0001_00_000002 completed successfully
2016-05-17 22:18:28,977 INFO impl.DAGImpl (DAGImpl.java:commitOrAbortOutputs(871)) - No output committers for vertex: vertex_1463523506869_0001_1_00 [E205BB9BAA7744F9B0E4DC91DE388B4F]
2016-05-17 22:18:28,984 INFO recovery.RecoveryService (RecoveryService.java:handle(306)) - DAG completed, dagId=dag_1463523506869_0001_1, queueSize=0
2016-05-17 22:18:28,985 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event DAG_FINISHED to history file
2016-05-17 22:18:28,987 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1463523506869_0001_1][Event:DAG_FINISHED]: dagId=dag_1463523506869_0001_1, startTime=1463523507996, finishTime=1463523508977, timeTaken=981, status=SUCCEEDED, diagnostics=, counters=Counters: 55, org.apache.tez.common.counters.DAGCounter, NUM_SUCCEEDED_TASKS=2, TOTAL_LAUNCHED_TASKS=2, RACK_LOCAL_TASKS=1, File System Counters, FILE_BYTES_READ=317, FILE_BYTES_WRITTEN=791, FILE_READ_OPS=0, FILE_LARGE_READ_OPS=0, FILE_WRITE_OPS=0, org.apache.tez.common.counters.TaskCounter, REDUCE_INPUT_GROUPS=4, REDUCE_INPUT_RECORDS=13, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=26, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, GC_TIME_MILLIS=0, CPU_MILLISECONDS=0, PHYSICAL_MEMORY_BYTES=0, VIRTUAL_MEMORY_BYTES=0, COMMITTED_HEAP_BYTES=590348288, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=26, OUTPUT_BYTES=195, OUTPUT_BYTES_WITH_OVERHEAD=171, OUTPUT_BYTES_PHYSICAL=175, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=175, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_BYTES=175, SHUFFLE_BYTES_DECOMPRESSED=171, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=175, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=32, MERGE_PHASE_TIME=51, FIRST_EVENT_RECEIVED=26, LAST_EVENT_RECEIVED=26, Shuffle Errors, BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0, cascading.flow.SliceCounters, Process_Begin_Time=2927047017586, Process_Duration=27, Process_End_Time=2927047017613, Read_Duration=4, Tuples_Read=26, Tuples_Written=26, Write_Duration=2, cascading.flow.StepCounters, Tuples_Read=13, Tuples_Written=13
2016-05-17 22:18:28,988 INFO impl.DAGImpl (DAGImpl.java:finished(1180)) - DAG: dag_1463523506869_0001_1 finished with state: SUCCEEDED
2016-05-17 22:18:28,989 INFO impl.DAGImpl (DAGImpl.java:handle(973)) - dag_1463523506869_0001_1 transitioned from RUNNING to SUCCEEDED due to event DAG_VERTEX_COMPLETED
2016-05-17 22:18:28,989 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:stop(242)) - Ignoring stop request for containerId: container_1463523506869_0001_00_000002
2016-05-17 22:18:28,990 INFO container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1463523506869_0001_00_000002 transitioned from STOP_REQUESTED to COMPLETED via event C_COMPLETED
2016-05-17 22:18:28 Completed Dag: dag_1463523506869_0001_1
2016-05-17 22:18:28,991 INFO app.DAGAppMaster (DAGAppMaster.java:handle(579)) - DAG completed, dagId=dag_1463523506869_0001_1, dagState=SUCCEEDED
2016-05-17 22:18:28,992 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(164)) - Redirecting log file based on addend: dag_1463523506869_0001_1_post
2016-05-17 22:18:28,992 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(179)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-17 22:18:28,993 INFO app.DAGAppMaster (DAGAppMaster.java:handle(625)) - Waiting for next DAG to be submitted.
2016-05-17 22:18:29,000 INFO client.TezClient (TezClient.java:stop(495)) - Shutting down Tez Session, sessionName=(1/1) ...stIdentityBuffer/identity, applicationId=application_1463523506869_0001
2016-05-17 22:18:29,005 INFO client.DAGClientHandler (DAGClientHandler.java:shutdownAM(117)) - Received message to shutdown AM
2016-05-17 22:18:29,005 INFO rm.TaskSchedulerEventHandler (TaskSchedulerEventHandler.java:setShouldUnregisterFlag(580)) - TaskScheduler notified that it should unregister from RM
2016-05-17 22:18:29,005 INFO app.DAGAppMaster (DAGAppMaster.java:shutdownTezAM(1088)) - No current running DAG, shutting down the AM
2016-05-17 22:18:29,005 INFO app.DAGAppMaster (DAGAppMaster.java:shutdown(692)) - DAGAppMasterShutdownHandler invoked
2016-05-17 22:18:29,006 INFO app.DAGAppMaster (DAGAppMaster.java:shutdown(701)) - Handling DAGAppMaster shutdown
2016-05-17 22:18:29,007 INFO app.DAGAppMaster (DAGAppMaster.java:run(721)) - Sleeping for 5 seconds before shutting down
2016-05-17 22:18:29,010 INFO history.HistoryEventHandler (HistoryEventHandler.java:serviceStop(79)) - Stopping HistoryEventHandler
2016-05-17 22:18:29,010 INFO recovery.RecoveryService (RecoveryService.java:serviceStop(189)) - Stopping RecoveryService
2016-05-17 22:18:29,010 INFO recovery.RecoveryService (RecoveryService.java:serviceStop(192)) - Handle the remaining events in queue, queue size=0
2016-05-17 22:18:29,011 INFO recovery.RecoveryService (RecoveryService.java:run(166)) - EventQueue take interrupted. Returning
2016-05-17 22:18:29,011 INFO recovery.RecoveryService (RecoveryService.java:serviceStop(214)) - Closing Summary Stream
2016-05-17 22:18:29,012 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:serviceStop(118)) - Stopping SimpleHistoryLoggingService, eventQueueBacklog=0
2016-05-17 22:18:29,012 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:run(105)) - EventQueue take interrupted. Returning
2016-05-17 22:18:29,017 INFO app.DAGAppMaster (DAGAppMaster.java:run(1773)) - Completed deletion of tez scratch data dir, path=file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.5.x-tez-0.6.x/build/test/tmp/.staging/A69585C002A74BF4ABAEEC706FE1A188/A9088A6DF9614808BCA4DF4454CB724E/.tez/application_1463523506869_0001
2016-05-17 22:18:29,019 INFO util.Hadoop18TapUtil (Hadoop18TapUtil.java:cleanTempPath(226)) - deleting temp path /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.5.x-tez-0.6.x/build/test/output/hadoop2-tez/bufferpipesplatform/testIdentityBuffer/identity/_temporary
2016-05-17 22:18:29,020 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [test] completed in: 00:02.356
2016-05-17 22:18:29,112 INFO cascading.PlatformTestCase (PlatformTestCase.java:tearDown(189)) - copying to local /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.5.x-tez-0.6.x/build/test/output//hadoop2-tez/bufferpipesplatform/testIdentityBuffer/identity
2016-05-17 22:18:29,149 INFO util.Util (Util.java:findMainClass(1374)) - using default application jar, may cause class not found exceptions on the cluster
2016-05-17 22:18:29,150 INFO planner.Hadoop2TezPlanner (Hadoop2TezPlanner.java:initialize(151)) - using application jar: /opt/concurrent/home/teamcity/.gradle/caches/modules-2/files-2.1/cascading/cascading-hadoop2-tez/3.1.0/226af02e24d5195c60e4ebe911d0c7c9780f59f0/cascading-hadoop2-tez-3.1.0.jar
2016-05-17 22:18:29,223 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [test] executed rule registry: NoHashJoinHadoop2TezRuleRegistry, completed as: SUCCESS, in: 00:00.036
2016-05-17 22:18:29,249 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [test] executed rule registry: HashJoinHadoop2TezRuleRegistry, completed as: SUCCESS, in: 00:00.062
2016-05-17 22:18:29,250 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [test] rule registry: NoHashJoinHadoop2TezRuleRegistry, supports assembly with steps: 1, nodes: 2
2016-05-17 22:18:29,250 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [test] rule registry: HashJoinHadoop2TezRuleRegistry, supports assembly with steps: 1, nodes: 2
2016-05-17 22:18:29,251 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [test] rule registry: NoHashJoinHadoop2TezRuleRegistry, result was selected using: 'default comparator: selects plan with fewest steps and fewest nodes'
2016-05-17 22:18:29,285 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [test] tap forcing step to tez local mode: /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.5.x-tez-0.6.x/build/classes/test/data/lhs.txt
2016-05-17 22:18:29,315 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [test] tap forcing step to tez local mode: /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.5.x-tez-0.6.x/build/test/output/hadoop2-tez/bufferpipesplatform/testFirstNBufferForward/firstn
2016-05-17 22:18:29,409 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [test] starting
2016-05-17 22:18:29,410 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [test] source: Hfs["TextDelimited[['num', 'lower']]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.5.x-tez-0.6.x/build/classes/test/data/lhs.txt"]
2016-05-17 22:18:29,411 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [test] sink: Hfs["TextDelimited[['num', 'lower']]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.5.x-tez-0.6.x/build/test/output/hadoop2-tez/bufferpipesplatform/testFirstNBufferForward/firstn"]
2016-05-17 22:18:29,412 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [test] parallel execution of steps is enabled: false
2016-05-17 22:18:29,412 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [test] executing total steps: 1
2016-05-17 22:18:29,413 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [test] allocating management threads: 1
2016-05-17 22:18:29,414 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [test] starting step: (1/1) ...irstNBufferForward/firstn
2016-05-17 22:18:29,415 WARN flow.Flow (BaseFlow.java:logWarn(1516)) - [test] 'yarn.timeline-service.enabled' is disabled, please enable to capture detailed metrics of completed flows, this may require starting the YARN timeline server daemon
2016-05-17 22:18:29,436 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [test] tez session mode enabled: true
2016-05-17 22:18:29,442 INFO client.TezClient (TezClient.java:<init>(153)) - Tez Client Version: [ component=tez-api, version=0.6.2, revision=c04283bf91fd96823eec37056d5c8b1a92728265, SCM-URL=scm:git:https://git-wip-us.apache.org/repos/asf/tez.git, buildTime=20150804-1436 ]
2016-05-17 22:18:29,443 INFO client.TezClient (TezClient.java:start(333)) - Session mode. Starting session.
2016-05-17 22:18:29,443 INFO client.TezClientUtils (TezClientUtils.java:setupTezJarsLocalResources(159)) - Ignoring 'tez.lib.uris' since 'tez.ignore.lib.uris' is set to true
2016-05-17 22:18:29,454 INFO client.TezClient (TezCommonUtils.java:createTezSystemStagingPath(118)) - Tez system stage directory file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.5.x-tez-0.6.x/build/test/tmp/.staging/D4BA3B0E396F46B28385B75D34E9E6FB/A57EE8E6A30E41329B7BA8BAFBA4EAD2/.tez/application_1463523509443_0001 doesn't exist and is created
2016-05-17 22:18:29,484 INFO client.LocalClient (LocalClient.java:createDAGAppMaster(328)) - DAGAppMaster thread has been created
2016-05-17 22:18:29,484 INFO client.LocalClient (LocalClient.java:startDAGAppMaster(251)) - DAGAppMaster is not created wait for 100ms...
2016-05-17 22:18:29,486 INFO client.LocalClient (LocalClient.java:run(275)) - Using staging directory: /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.5.x-tez-0.6.x/build/test/tmp/.staging/D4BA3B0E396F46B28385B75D34E9E6FB/A57EE8E6A30E41329B7BA8BAFBA4EAD2/.tez/application_1463523509443_0001
2016-05-17 22:18:29,490 INFO app.DAGAppMaster (DAGAppMaster.java:<init>(309)) - Created DAGAppMaster for application appattempt_1463523509443_0001_000000, versionInfo=[ component=tez-dag, version=0.6.2, revision=c04283bf91fd96823eec37056d5c8b1a92728265, SCM-URL=scm:git:https://git-wip-us.apache.org/repos/asf/tez.git, buildTime=20150804-1436 ]
2016-05-17 22:18:29,511 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(333)) - Comparing client version with AM version, clientVersion=0.6.2, AMVersion=0.6.2
2016-05-17 22:18:29,512 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(386)) - Adding session token to jobTokenSecretManager for application
2016-05-17 22:18:29,514 INFO node.AMNodeTracker (AMNodeTracker.java:serviceInit(77)) - blacklistDisablePercent is 33, blacklistingEnabled: false, maxTaskFailuresPerNode: 1
2016-05-17 22:18:29,514 INFO history.HistoryEventHandler (HistoryEventHandler.java:serviceInit(50)) - Initializing HistoryEventHandler
2016-05-17 22:18:29,514 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:serviceInit(66)) - Log file location for SimpleHistoryLoggingService not specified, defaulting to containerLogDir=/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.5.x-tez-0.6.x/build/test/tmp/.staging/D4BA3B0E396F46B28385B75D34E9E6FB/A57EE8E6A30E41329B7BA8BAFBA4EAD2/.tez/application_1463523509443_0001/localmode-log-dir
2016-05-17 22:18:29,515 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:serviceInit(88)) - Initializing SimpleHistoryLoggingService, logFileLocation=/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.5.x-tez-0.6.x/build/test/tmp/.staging/D4BA3B0E396F46B28385B75D34E9E6FB/A57EE8E6A30E41329B7BA8BAFBA4EAD2/.tez/application_1463523509443_0001/localmode-log-dir/history.txt.appattempt_1463523509443_0001_000000, maxErrors=10
2016-05-17 22:18:29,515 INFO recovery.RecoveryService (RecoveryService.java:serviceInit(109)) - Initializing RecoveryService
2016-05-17 22:18:29,515 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:N/A][Event:AM_LAUNCHED]: appAttemptId=appattempt_1463523509443_0001_000000, appSubmitTime=1463523509490, launchTime=1463523509490
2016-05-17 22:18:29,523 INFO history.HistoryEventHandler (HistoryEventHandler.java:serviceStart(73)) - Starting HistoryEventHandler
2016-05-17 22:18:29,529 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:serviceStart(95)) - Starting SimpleHistoryLoggingService
2016-05-17 22:18:29,532 INFO recovery.RecoveryService (RecoveryService.java:serviceStart(127)) - Starting RecoveryService
2016-05-17 22:18:29,532 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event AM_LAUNCHED to history file
2016-05-17 22:18:29,536 INFO client.DAGClientServer (DAGClientServer.java:serviceStart(91)) - Instantiated DAGClientRPCServer at ip-10-123-148-208.ec2.internal/10.123.148.208:38186
2016-05-17 22:18:29,540 WARN impl.MetricsConfig (MetricsConfig.java:loadFirst(124)) - Cannot locate configuration: tried hadoop-metrics2-dagappmaster.properties,hadoop-metrics2.properties
2016-05-17 22:18:29,542 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event AM_STARTED to history file
2016-05-17 22:18:29,542 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:N/A][Event:AM_STARTED]: appAttemptId=appattempt_1463523509443_0001_000000, startTime=1463523509542
2016-05-17 22:18:29,543 INFO app.DAGAppMaster (DAGAppMaster.java:serviceStart(1644)) - In Session mode. Waiting for DAG over RPC
2016-05-17 22:18:29,585 INFO client.LocalClient (LocalClient.java:startDAGAppMaster(236)) - DAGAppMaster state: IDLE
2016-05-17 22:18:29,585 INFO client.TezClient (TezClient.java:start(367)) - The url to track the Tez Session: N/A
2016-05-17 22:18:29,586 INFO client.TezClient (TezClient.java:submitDAGSession(405)) - Submitting dag to TezSession, sessionName=(1/1) ...irstNBufferForward/firstn, applicationId=application_1463523509443_0001, dagName=[D4BA3B0E396F46B28385B75D34E9E6FB/A57EE8E6A30E41329B7BA8BAFBA4EAD2] test/(1/1) ...irstNBufferForward/firstn
2016-05-17 22:18:29,645 INFO app.DAGAppMaster (DAGAppMaster.java:submitDAGToAppMaster(1126)) - Starting DAG submitted via RPC: [D4BA3B0E396F46B28385B75D34E9E6FB/A57EE8E6A30E41329B7BA8BAFBA4EAD2] test/(1/1) ...irstNBufferForward/firstn
2016-05-17 22:18:29,650 INFO app.DAGAppMaster (DAGAppMaster.java:generateDAGVizFile(870)) - Generating DAG graphviz file, dagId=dag_1463523509443_0001_1, filePath=/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.5.x-tez-0.6.x/build/test/tmp/.staging/D4BA3B0E396F46B28385B75D34E9E6FB/A57EE8E6A30E41329B7BA8BAFBA4EAD2/.tez/application_1463523509443_0001/localmode-log-dir/dag_1463523509443_0001_1.dot
2016-05-17 22:18:29,651 INFO app.DAGAppMaster (DAGAppMaster.java:writePBTextFile(887)) - Writing DAG plan to: /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.5.x-tez-0.6.x/build/test/tmp/.staging/D4BA3B0E396F46B28385B75D34E9E6FB/A57EE8E6A30E41329B7BA8BAFBA4EAD2/.tez/application_1463523509443_0001/localmode-log-dir/dag_1463523509443_0001_1-tez-dag.pb.txt
2016-05-17 22:18:29,678 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(164)) - Redirecting log file based on addend: dag_1463523509443_0001_1
2016-05-17 22:18:29,678 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(179)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-17 22:18:29,678 INFO app.DAGAppMaster (DAGAppMaster.java:startDAG(2051)) - Running DAG: [D4BA3B0E396F46B28385B75D34E9E6FB/A57EE8E6A30E41329B7BA8BAFBA4EAD2] test/(1/1) ...irstNBufferForward/firstn
2016-05-17 22:18:29 Running Dag: dag_1463523509443_0001_1
2016-05-17 22:18:29,694 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1463523509443_0001_1][Event:DAG_SUBMITTED]: dagID=dag_1463523509443_0001_1, submitTime=1463523509646
2016-05-17 22:18:29,694 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event DAG_SUBMITTED to history file
2016-05-17 22:18:29,695 INFO impl.VertexImpl (VertexImpl.java:setAdditionalOutputs(4033)) - setting additional outputs for vertex C3F9661B4D0043369F8663BCAEB33DFE
2016-05-17 22:18:29,696 INFO impl.DAGImpl (DAGImpl.java:assignDAGScheduler(1381)) - Using DAG Scheduler: org.apache.tez.dag.app.dag.impl.DAGSchedulerNaturalOrderControlled
2016-05-17 22:18:29,696 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event DAG_INITIALIZED to history file
2016-05-17 22:18:29,697 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1463523509443_0001_1][Event:DAG_INITIALIZED]: dagID=dag_1463523509443_0001_1, initTime=1463523509694
2016-05-17 22:18:29,697 INFO impl.DAGImpl (DAGImpl.java:handle(973)) - dag_1463523509443_0001_1 transitioned from NEW to INITED due to event DAG_INIT
2016-05-17 22:18:29,698 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1463523509443_0001_1][Event:DAG_STARTED]: dagID=dag_1463523509443_0001_1, startTime=1463523509698
2016-05-17 22:18:29,698 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event DAG_STARTED to history file
2016-05-17 22:18:29,698 INFO impl.DAGImpl (DAGImpl.java:transition(1614)) - Added additional resources : [[]] to classpath
2016-05-17 22:18:29,699 INFO impl.DAGImpl (DAGImpl.java:handle(973)) - dag_1463523509443_0001_1 transitioned from INITED to RUNNING due to event DAG_START
2016-05-17 22:18:29,699 INFO impl.VertexImpl (VertexImpl.java:setupVertex(2068)) - Root Inputs exist for Vertex: A66EEB93ECE5442F8F1D1249181D9A6C : {BB8A698F4FD141B9A6D0905721F2598E={InputName=BB8A698F4FD141B9A6D0905721F2598E}, {Descriptor=ClassName=org.apache.tez.mapreduce.input.MRInput, hasPayload=true}, {ControllerDescriptor=ClassName=org.apache.tez.mapreduce.common.MRInputAMSplitGenerator, hasPayload=false}}
2016-05-17 22:18:29,699 INFO impl.VertexImpl (VertexImpl.java:setupVertex(2078)) - Starting root input initializer for input: BB8A698F4FD141B9A6D0905721F2598E, with class: [org.apache.tez.mapreduce.common.MRInputAMSplitGenerator]
2016-05-17 22:18:29,699 INFO impl.VertexImpl (VertexImpl.java:assignVertexManager(2191)) - Setting vertexManager to RootInputVertexManager for vertex_1463523509443_0001_1_00 [A66EEB93ECE5442F8F1D1249181D9A6C]
2016-05-17 22:18:29,700 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(2983)) - Num tasks is -1. Expecting VertexManager/InputInitializers/1-1 split to set #tasks for the vertex vertex_1463523509443_0001_1_00 [A66EEB93ECE5442F8F1D1249181D9A6C]
2016-05-17 22:18:29,700 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(2987)) - Vertex will initialize from input initializer. vertex_1463523509443_0001_1_00 [A66EEB93ECE5442F8F1D1249181D9A6C]
2016-05-17 22:18:29,700 INFO client.TezClient (TezClient.java:submitDAGSession(478)) - Submitted dag to TezSession, sessionName=(1/1) ...irstNBufferForward/firstn, applicationId=application_1463523509443_0001, dagName=[D4BA3B0E396F46B28385B75D34E9E6FB/A57EE8E6A30E41329B7BA8BAFBA4EAD2] test/(1/1) ...irstNBufferForward/firstn
2016-05-17 22:18:29,700 INFO impl.VertexImpl (VertexImpl.java:setupInputInitializerManager(3933)) - Vertex will initialize via inputInitializers vertex_1463523509443_0001_1_00 [A66EEB93ECE5442F8F1D1249181D9A6C]. Starting root input initializers: 1
2016-05-17 22:18:29,701 INFO impl.VertexImpl (VertexImpl.java:handle(1564)) - vertex_1463523509443_0001_1_00 [A66EEB93ECE5442F8F1D1249181D9A6C] transitioned from NEW to INITIALIZING due to event V_INIT
2016-05-17 22:18:29,701 INFO impl.VertexImpl (VertexImpl.java:assignVertexManager(2203)) - Setting vertexManager to ShuffleVertexManager for vertex_1463523509443_0001_1_01 [C3F9661B4D0043369F8663BCAEB33DFE]
2016-05-17 22:18:29,703 INFO dag.RootInputInitializerManager (RootInputInitializerManager.java:run(242)) - Starting InputInitializer for Input: BB8A698F4FD141B9A6D0905721F2598E on vertex vertex_1463523509443_0001_1_00 [A66EEB93ECE5442F8F1D1249181D9A6C]
2016-05-17 22:18:29,710 INFO common.MRInputAMSplitGenerator (MRInputAMSplitGenerator.java:initialize(92)) - Emitting serialized splits: true
2016-05-17 22:18:29,711 INFO common.MRInputAMSplitGenerator (MRInputAMSplitGenerator.java:initialize(110)) - Input BB8A698F4FD141B9A6D0905721F2598E asking for 1 tasks. Headroom: 672 Task Resource: 1024 waves: 1.7
2016-05-17 22:18:29,713 INFO common.MRInputAMSplitGenerator (MRInputAMSplitGenerator.java:initialize(121)) - Grouping input splits
2016-05-17 22:18:29,714 INFO hadoop.MRInputHelpers (MRInputHelpers.java:generateInputSplitsToMem(294)) - Generating mapred api input splits
2016-05-17 22:18:29,717 INFO hadoop.MRInputHelpers (MRInputHelpers.java:generateInputSplitsToMem(300)) - NumSplits: 1, SerializedSize: 396
2016-05-17 22:18:29,719 INFO dag.RootInputInitializerManager (RootInputInitializerManager.java:onSuccess(272)) - Succeeded InputInitializer for Input: BB8A698F4FD141B9A6D0905721F2598E on vertex vertex_1463523509443_0001_1_00 [A66EEB93ECE5442F8F1D1249181D9A6C]
2016-05-17 22:18:29,721 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:initialize(696)) - Shuffle Vertex Manager: settings minFrac:0.25 maxFrac:0.75 auto:false desiredTaskIput:104857600 minTasks:1
2016-05-17 22:18:29,722 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(3014)) - Creating 1 tasks for vertex: vertex_1463523509443_0001_1_01 [C3F9661B4D0043369F8663BCAEB33DFE]
2016-05-17 22:18:29,723 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(3026)) - Directly initializing vertex: vertex_1463523509443_0001_1_01 [C3F9661B4D0043369F8663BCAEB33DFE]
2016-05-17 22:18:29,724 INFO impl.VertexImpl (VertexImpl.java:initializeCommitters(1915)) - Invoking committer inits for vertex, vertexId=vertex_1463523509443_0001_1_01 [C3F9661B4D0043369F8663BCAEB33DFE]
2016-05-17 22:18:29,724 INFO impl.VertexImpl (VertexImpl.java:initializeCommitters(1927)) - Instantiating committer for output=91DD59C39FB448D58BDA96BBC66DE830, vertexId=vertex_1463523509443_0001_1_01 [C3F9661B4D0043369F8663BCAEB33DFE], committerClass=org.apache.tez.mapreduce.committer.MROutputCommitter
2016-05-17 22:18:29,725 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [test] submitted tez dag to app master: application_1463523509443_0001, with dag id: dag_1463523509443_0001_1
2016-05-17 22:18:29,725 INFO impl.VertexImpl (VertexImpl.java:run(1945)) - Invoking committer init for output=91DD59C39FB448D58BDA96BBC66DE830, vertexId=vertex_1463523509443_0001_1_01 [C3F9661B4D0043369F8663BCAEB33DFE]
2016-05-17 22:18:29,730 INFO committer.MROutputCommitter (MROutputCommitter.java:getOutputCommitter(145)) - OutputCommitter for outputName=91DD59C39FB448D58BDA96BBC66DE830, vertexName=C3F9661B4D0043369F8663BCAEB33DFE, outputCommitterClass=org.apache.hadoop.mapred.FileOutputCommitter
2016-05-17 22:18:29,731 INFO impl.VertexImpl (VertexImpl.java:run(1949)) - Invoking committer setup for output=91DD59C39FB448D58BDA96BBC66DE830, vertexId=vertex_1463523509443_0001_1_01 [C3F9661B4D0043369F8663BCAEB33DFE]
2016-05-17 22:18:29,733 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event VERTEX_INITIALIZED to history file
2016-05-17 22:18:29,733 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1463523509443_0001_1][Event:VERTEX_INITIALIZED]: vertexName=C3F9661B4D0043369F8663BCAEB33DFE, vertexId=vertex_1463523509443_0001_1_01, initRequestedTime=1463523509701, initedTime=1463523509733, numTasks=1, processorName=cascading.flow.tez.FlowProcessor, additionalInputsCount=0
2016-05-17 22:18:29,734 INFO impl.VertexImpl (VertexImpl.java:handle(1564)) - vertex_1463523509443_0001_1_01 [C3F9661B4D0043369F8663BCAEB33DFE] transitioned from NEW to INITED due to event V_INIT
2016-05-17 22:18:29,735 INFO impl.VertexImpl (VertexImpl.java:canInitVertex(3195)) - Cannot init vertex: vertex_1463523509443_0001_1_00 [A66EEB93ECE5442F8F1D1249181D9A6C] numTasks: -1 numUnitializedEdges: 0 numInitializedInputs: 0 initWaitsForRootInitializers: true
2016-05-17 22:18:29,735 INFO impl.VertexImpl (VertexImpl.java:setParallelism(1374)) - Got updated RootInputsSpecs: {BB8A698F4FD141B9A6D0905721F2598E=forAllWorkUnits=true, update=[1]}
2016-05-17 22:18:29,736 INFO impl.VertexImpl (VertexImpl.java:setParallelism(1393)) - Vertex vertex_1463523509443_0001_1_00 [A66EEB93ECE5442F8F1D1249181D9A6C] parallelism set to 1
2016-05-17 22:18:29,736 INFO impl.VertexImpl (VertexImpl.java:canInitVertex(3195)) - Cannot init vertex: vertex_1463523509443_0001_1_00 [A66EEB93ECE5442F8F1D1249181D9A6C] numTasks: 1 numUnitializedEdges: 0 numInitializedInputs: 0 initWaitsForRootInitializers: true
2016-05-17 22:18:29,738 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1463523509443_0001_1][Event:VERTEX_DATA_MOVEMENT_EVENTS_GENERATED]: vertexId=vertex_1463523509443_0001_1_00, eventCount=1
2016-05-17 22:18:29,738 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event VERTEX_INITIALIZED to history file
2016-05-17 22:18:29,739 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1463523509443_0001_1][Event:VERTEX_INITIALIZED]: vertexName=A66EEB93ECE5442F8F1D1249181D9A6C, vertexId=vertex_1463523509443_0001_1_00, initRequestedTime=1463523509699, initedTime=1463523509738, numTasks=1, processorName=cascading.flow.tez.FlowProcessor, additionalInputsCount=1
2016-05-17 22:18:29,740 INFO impl.VertexImpl (VertexImpl.java:startIfPossible(3061)) - Triggering start event for vertex: vertex_1463523509443_0001_1_00 [A66EEB93ECE5442F8F1D1249181D9A6C] with distanceFromRoot: 0
2016-05-17 22:18:29,740 INFO impl.VertexImpl (VertexImpl.java:handle(1564)) - vertex_1463523509443_0001_1_00 [A66EEB93ECE5442F8F1D1249181D9A6C] transitioned from INITIALIZING to INITED due to event V_ROOT_INPUT_INITIALIZED
2016-05-17 22:18:29,742 INFO impl.ImmediateStartVertexManager (ImmediateStartVertexManager.java:scheduleTasks(99)) - Starting 1 in A66EEB93ECE5442F8F1D1249181D9A6C
2016-05-17 22:18:29,743 INFO impl.VertexImpl (VertexImpl.java:scheduleTasks(1250)) - Routing pending task events for vertex: vertex_1463523509443_0001_1_00 [A66EEB93ECE5442F8F1D1249181D9A6C]
2016-05-17 22:18:29,744 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event VERTEX_STARTED to history file
2016-05-17 22:18:29,745 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1463523509443_0001_1][Event:VERTEX_STARTED]: vertexId=vertex_1463523509443_0001_1_00, startRequestedTime=1463523509742, startedTime=1463523509742
2016-05-17 22:18:29,745 INFO impl.VertexImpl (VertexImpl.java:handle(1564)) - vertex_1463523509443_0001_1_00 [A66EEB93ECE5442F8F1D1249181D9A6C] transitioned from INITED to RUNNING due to event V_START
2016-05-17 22:18:29,746 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event TASK_STARTED to history file
2016-05-17 22:18:29,747 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1463523509443_0001_1][Event:TASK_STARTED]: vertexName=A66EEB93ECE5442F8F1D1249181D9A6C, taskId=task_1463523509443_0001_1_00_000000, scheduledTime=1463523509746, launchTime=1463523509746
2016-05-17 22:18:29,747 INFO impl.TaskImpl (TaskImpl.java:handle(862)) - task_1463523509443_0001_1_00_000000 Task Transitioned from NEW to SCHEDULED due to event T_SCHEDULE
2016-05-17 22:18:29,747 INFO impl.VertexImpl (VertexImpl.java:transition(3155)) - Source vertex started: vertex_1463523509443_0001_1_00 for vertex: vertex_1463523509443_0001_1_01 [C3F9661B4D0043369F8663BCAEB33DFE] numStartedSources: 1 numSources: 1
2016-05-17 22:18:29,748 INFO impl.VertexImpl (VertexImpl.java:startIfPossible(3061)) - Triggering start event for vertex: vertex_1463523509443_0001_1_01 [C3F9661B4D0043369F8663BCAEB33DFE] with distanceFromRoot: 1
2016-05-17 22:18:29,748 INFO impl.DAGSchedulerNaturalOrderControlled (DAGSchedulerNaturalOrderControlled.java:scheduleTask(111)) - Scheduled vertex: vertex_1463523509443_0001_1_00 [A66EEB93ECE5442F8F1D1249181D9A6C]
2016-05-17 22:18:29,749 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:handleVertexStateUpdate(720)) - Received configured notification : CONFIGURED for vertex: A66EEB93ECE5442F8F1D1249181D9A6C in vertex: C3F9661B4D0043369F8663BCAEB33DFE
2016-05-17 22:18:29,749 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:onVertexStarted(355)) - OnVertexStarted vertex: C3F9661B4D0043369F8663BCAEB33DFE with 1 source tasks and 1 pending tasks
2016-05-17 22:18:29,750 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event VERTEX_STARTED to history file
2016-05-17 22:18:29,750 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1463523509443_0001_1][Event:VERTEX_STARTED]: vertexId=vertex_1463523509443_0001_1_01, startRequestedTime=1463523509748, startedTime=1463523509748
2016-05-17 22:18:29,750 INFO impl.VertexImpl (VertexImpl.java:handle(1564)) - vertex_1463523509443_0001_1_01 [C3F9661B4D0043369F8663BCAEB33DFE] transitioned from INITED to RUNNING due to event V_START
2016-05-17 22:18:29,751 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:transition(1014)) - remoteTaskSpec:DAGName : [D4BA3B0E396F46B28385B75D34E9E6FB/A57EE8E6A30E41329B7BA8BAFBA4EAD2] test/(1/1) ...irstNBufferForward/firstn, VertexName: A66EEB93ECE5442F8F1D1249181D9A6C, VertexParallelism: 1, TaskAttemptID:attempt_1463523509443_0001_1_00_000000_0, processorName=cascading.flow.tez.FlowProcessor, inputSpecListSize=1, outputSpecListSize=1, inputSpecList=[{{ sourceVertexName=BB8A698F4FD141B9A6D0905721F2598E, physicalEdgeCount=1, inputClassName=org.apache.tez.mapreduce.input.MRInput }}, ], outputSpecList=[{{ destinationVertexName=C3F9661B4D0043369F8663BCAEB33DFE, physicalEdgeCount=1, outputClassName=org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput }}, ]
2016-05-17 22:18:29,752 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:handle(692)) - attempt_1463523509443_0001_1_00_000000_0 TaskAttempt Transitioned from NEW to START_WAIT due to event TA_SCHEDULE
2016-05-17 22:18:29,752 INFO rm.TaskSchedulerEventHandler (TaskSchedulerEventHandler.java:handleEvent(149)) - Processing the event EventType: S_TA_LAUNCH_REQUEST
2016-05-17 22:18:29,753 INFO node.AMNodeTracker (AMNodeTracker.java:nodeSeen(91)) - Adding new node: 127.0.0.1:0
2016-05-17 22:18:29,754 INFO container.AMContainerHelpers (AMContainerHelpers.java:createCommonContainerLaunchContext(112)) - Adding #1 tokens and #0 secret keys for NM use for launching container
2016-05-17 22:18:29,754 INFO container.AMContainerHelpers (AMContainerHelpers.java:createCommonContainerLaunchContext(122)) - Putting shuffle token in serviceData
2016-05-17 22:18:29,755 INFO container.AMContainerImpl (AMContainerImpl.java:transition(521)) - Sending Launch Request for Container with id: container_1463523509443_0001_00_000001
2016-05-17 22:18:29,755 INFO container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1463523509443_0001_00_000001 transitioned from ALLOCATED to LAUNCHING via event C_LAUNCH_REQUEST
2016-05-17 22:18:29,758 INFO task.TezChild (TezChild.java:newTezChild(432)) - PID, containerIdentifier: null, container_1463523509443_0001_00_000001
2016-05-17 22:18:29,758 WARN impl.MetricsSystemImpl (MetricsSystemImpl.java:init(145)) - DAGAppMaster metrics system already initialized!
2016-05-17 22:18:29,759 INFO container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1463523509443_0001_00_000001 transitioned from LAUNCHING to IDLE via event C_LAUNCHED
2016-05-17 22:18:29,760 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event CONTAINER_LAUNCHED to history file
2016-05-17 22:18:29,760 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1463523509443_0001_1][Event:CONTAINER_LAUNCHED]: containerId=container_1463523509443_0001_00_000001, launchTime=1463523509759
2016-05-17 22:18:29,761 INFO task.ContainerReporter (ContainerReporter.java:call(55)) - Attempting to fetch new task for container container_1463523509443_0001_00_000001
2016-05-17 22:18:29,762 INFO container.AMContainerImpl (AMContainerImpl.java:transition(827)) - Assigned taskAttempt + [attempt_1463523509443_0001_1_00_000000_0] to container: [container_1463523509443_0001_00_000001]
2016-05-17 22:18:29,762 INFO container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1463523509443_0001_00_000001 transitioned from IDLE to RUNNING via event C_PULL_TA
2016-05-17 22:18:29,762 INFO app.TaskAttemptListenerImpTezDag (TaskAttemptListenerImpTezDag.java:getTask(240)) - Container with id: container_1463523509443_0001_00_000001 given task: attempt_1463523509443_0001_1_00_000000_0
2016-05-17 22:18:29,762 INFO task.ContainerReporter (ContainerReporter.java:call(65)) - Got TaskUpdate for containerId= container_1463523509443_0001_00_000001: 0 ms after starting to poll. TaskInfo: shouldDie: false, currentTaskAttemptId: attempt_1463523509443_0001_1_00_000000_0
2016-05-17 22:18:29 Starting to run new task attempt: attempt_1463523509443_0001_1_00_000000_0
2016-05-17 22:18:29,763 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:transition(1146)) - TaskAttempt: [attempt_1463523509443_0001_1_00_000000_0] started. Is using containerId: [container_1463523509443_0001_00_000001] on NM: [127.0.0.1:0]
2016-05-17 22:18:29,763 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(164)) - Redirecting log file based on addend: attempt_1463523509443_0001_1_00_000000_0
2016-05-17 22:18:29,763 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(179)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-17 22:18:29,763 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event TASK_ATTEMPT_STARTED to history file
2016-05-17 22:18:29,763 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1463523509443_0001_1][Event:TASK_ATTEMPT_STARTED]: vertexName=A66EEB93ECE5442F8F1D1249181D9A6C, taskAttemptId=attempt_1463523509443_0001_1_00_000000_0, startTime=1463523509762, containerId=container_1463523509443_0001_00_000001, nodeId=127.0.0.1:0, inProgressLogs=127.0.0.1:0/node/containerlogs/container_1463523509443_0001_00_000001/teamcity, completedLogs=
2016-05-17 22:18:29,763 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:handle(692)) - attempt_1463523509443_0001_1_00_000000_0 TaskAttempt Transitioned from START_WAIT to RUNNING due to event TA_STARTED_REMOTELY
2016-05-17 22:18:29,764 INFO impl.TaskImpl (TaskImpl.java:handle(862)) - task_1463523509443_0001_1_00_000000 Task Transitioned from SCHEDULED to RUNNING due to event T_ATTEMPT_LAUNCHED
2016-05-17 22:18:29,764 INFO task.TezChild (TezChild.java:handleNewTaskCredentials(273)) - Refreshing UGI since Credentials have changed
2016-05-17 22:18:29,764 INFO task.TezChild (TezChild.java:handleNewTaskCredentials(276)) - Credentials : #Tokens=1, #SecretKeys=0
2016-05-17 22:18:29,764 INFO task.TezChild (TezChild.java:handleNewTaskLocalResources(302)) - Localizing additional local resources for Task : {}
2016-05-17 22:18:29,764 INFO task.TezChild (TezChild.java:handleNewTaskLocalResources(312)) - Done localizing additional resources
2016-05-17 22:18:29,765 INFO metrics.TaskCounterUpdater (TaskCounterUpdater.java:initResourceCalculatorPlugin(148)) - Using ResourceCalculatorProcessTree : [ ]
2016-05-17 22:18:29,765 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:<init>(149)) - Initializing LogicalIOProcessorRuntimeTask with TaskSpec: DAGName : [D4BA3B0E396F46B28385B75D34E9E6FB/A57EE8E6A30E41329B7BA8BAFBA4EAD2] test/(1/1) ...irstNBufferForward/firstn, VertexName: A66EEB93ECE5442F8F1D1249181D9A6C, VertexParallelism: 1, TaskAttemptID:attempt_1463523509443_0001_1_00_000000_0, processorName=cascading.flow.tez.FlowProcessor, inputSpecListSize=1, outputSpecListSize=1, inputSpecList=[{{ sourceVertexName=BB8A698F4FD141B9A6D0905721F2598E, physicalEdgeCount=1, inputClassName=org.apache.tez.mapreduce.input.MRInput }}, ], outputSpecList=[{{ destinationVertexName=C3F9661B4D0043369F8663BCAEB33DFE, physicalEdgeCount=1, outputClassName=org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput }}, ]
2016-05-17 22:18:29,765 INFO resources.MemoryDistributor (MemoryDistributor.java:<init>(85)) - InitialMemoryDistributor (isEnabled=true) invoked with: numInputs=1, numOutputs=1, JVM.maxFree=704643072
2016-05-17 22:18:29,766 INFO task.TezTaskRunner (TezTaskRunner.java:run(172)) - Initializing task, taskAttemptId=attempt_1463523509443_0001_1_00_000000_0
2016-05-17 22:18:29,767 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(191)) - Initializing LogicalProcessorIORuntimeTask
2016-05-17 22:18:29,768 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(195)) - Creating processor, processorClassName=cascading.flow.tez.FlowProcessor
2016-05-17 22:18:29,768 INFO impl.TaskImpl (TaskImpl.java:getTaskAttemptTezEvents(481)) - TaskAttempt:attempt_1463523509443_0001_1_00_000000_0 sent events: (0-1)
2016-05-17 22:18:29,769 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initializeLogicalIOProcessor(488)) - Initializing processor, processorClassName=cascading.flow.tez.FlowProcessor
2016-05-17 22:18:29,769 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(438)) - Initializing Output using OutputSpec: { destinationVertexName=C3F9661B4D0043369F8663BCAEB33DFE, physicalEdgeCount=1, outputClassName=org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput }
2016-05-17 22:18:29,769 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(389)) - Initializing Input using InputSpec: { sourceVertexName=BB8A698F4FD141B9A6D0905721F2598E, physicalEdgeCount=1, inputClassName=org.apache.tez.mapreduce.input.MRInput }
2016-05-17 22:18:29,770 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:createInput(537)) - Creating Input
2016-05-17 22:18:29,770 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:createOutput(565)) - Creating Output
2016-05-17 22:18:29,770 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(397)) - Initializing Input with src edge: BB8A698F4FD141B9A6D0905721F2598E
2016-05-17 22:18:29,771 INFO resources.MemoryDistributor$RequestorInfo (MemoryDistributor.java:<init>(233)) - Received request: 0, type: INPUT, componentVertexName: BB8A698F4FD141B9A6D0905721F2598E
2016-05-17 22:18:29,771 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(446)) - Initializing Output with dest edge: C3F9661B4D0043369F8663BCAEB33DFE
2016-05-17 22:18:29,777 INFO impl.ExternalSorter (ExternalSorter.java:getInitialMemoryRequirement(293)) - Requested SortBufferSize (tez.runtime.io.sort.mb): 100
2016-05-17 22:18:29,777 INFO resources.MemoryDistributor$RequestorInfo (MemoryDistributor.java:<init>(233)) - Received request: 104857600, type: OUTPUT, componentVertexName: C3F9661B4D0043369F8663BCAEB33DFE
2016-05-17 22:18:29,777 INFO input.MRInput (MRInput.java:initialize(440)) - Using New mapreduce API: false, split information via event: true
2016-05-17 22:18:29,779 INFO input.MRInput (MRInput.java:initializeInternal(493)) - Initialzed MRInput: BB8A698F4FD141B9A6D0905721F2598E
2016-05-17 22:18:29,780 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(403)) - Initialized Input with src edge: BB8A698F4FD141B9A6D0905721F2598E
2016-05-17 22:18:29,778 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(452)) - Initialized Output with dest edge: C3F9661B4D0043369F8663BCAEB33DFE
2016-05-17 22:18:29,779 INFO tez.FlowProcessor (FlowProcessor.java:initialize(88)) - cascading version: 3.1.0
2016-05-17 22:18:29,789 INFO tez.FlowProcessor (FlowProcessor.java:initialize(94)) - flow node id: A66EEB93ECE5442F8F1D1249181D9A6C, ordinal: 0
2016-05-17 22:18:29,789 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: A66EEB93ECE5442F8F1D1249181D9A6C, mem on start (mb), free: 138, total: 281, max: 672
2016-05-17 22:18:29,790 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initializeLogicalIOProcessor(491)) - Initialized processor, processorClassName=cascading.flow.tez.FlowProcessor
2016-05-17 22:18:29,791 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(220)) - Waiting for 2 initializers to finish
2016-05-17 22:18:29,791 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(220)) - Waiting for 1 initializers to finish
2016-05-17 22:18:29,791 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(233)) - All initializers finished
2016-05-17 22:18:29,791 INFO resources.MemoryDistributor (MemoryDistributor.java:makeInitialAllocations(124)) - Using Allocator class: org.apache.tez.runtime.library.resources.WeightedScalingMemoryDistributor
2016-05-17 22:18:29,792 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:getRequestTypeForClass(197)) - Falling back to RequestType.OTHER for class: org.apache.tez.mapreduce.input.MRInput
2016-05-17 22:18:29,792 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:initialProcessMemoryRequestContext(171)) - ScaleFactor: 1, for type: OTHER
2016-05-17 22:18:29,792 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:initialProcessMemoryRequestContext(171)) - ScaleFactor: 12, for type: SORTED_OUTPUT
2016-05-17 22:18:29,793 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:computeReservedFraction(262)) - InitialReservationFraction=0.3, AdditionalReservationFractionForIOs=0.03, finalReserveFractionUsed=0.32999999999999996
2016-05-17 22:18:29,794 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:assignMemory(126)) - Scaling Requests. NumRequests: 2, numScaledRequests: 13, TotalRequested: 104857600, TotalRequestedScaled: 9.679163076923077E7, TotalJVMHeap: 704643072, TotalAvailable: 472110858, TotalRequested/TotalJVMHeap:0.15
2016-05-17 22:18:29,795 INFO resources.MemoryDistributor (MemoryDistributor.java:makeInitialAllocations(138)) - Informing: INPUT, BB8A698F4FD141B9A6D0905721F2598E, org.apache.tez.mapreduce.input.MRInput: requested=0, allocated=0
2016-05-17 22:18:29,795 INFO resources.MemoryDistributor (MemoryDistributor.java:makeInitialAllocations(138)) - Informing: OUTPUT, C3F9661B4D0043369F8663BCAEB33DFE, org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput: requested=104857600, allocated=104857600
2016-05-17 22:18:29,795 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(255)) - Starting Inputs/Outputs
2016-05-17 22:18:29,795 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(268)) - Input: BB8A698F4FD141B9A6D0905721F2598E being auto started by the framework. Subsequent instances will not be auto-started
2016-05-17 22:18:29,796 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(289)) - Num IOs determined for AutoStart: 1
2016-05-17 22:18:29,796 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(291)) - Waiting for 1 IOs to start
2016-05-17 22:18:29,795 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(419)) - Starting Input with src edge: BB8A698F4FD141B9A6D0905721F2598E
2016-05-17 22:18:29,796 INFO input.MRInput (MRInput.java:start(450)) - MRInput setup to received events1
2016-05-17 22:18:29,796 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(421)) - Started Input with src edge: BB8A698F4FD141B9A6D0905721F2598E
2016-05-17 22:18:29,797 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(304)) - AutoStartComplete
2016-05-17 22:18:29,797 INFO task.TezTaskRunner (TezTaskRunner.java:run(175)) - Running task, taskAttemptId=attempt_1463523509443_0001_1_00_000000_0
2016-05-17 22:18:29,798 INFO input.MRInput (MRInput.java:initFromEventInternal(619)) - Initializing RecordReader from event
2016-05-17 22:18:29,802 INFO input.MRInput (MRInput.java:initFromEventInternal(630)) - Split Details -> SplitClass: org.apache.hadoop.mapred.split.TezGroupedSplit, OldSplit: org.apache.hadoop.mapred.split.TezGroupedSplit@3bc4d573
2016-05-17 22:18:29,804 INFO lib.MRReaderMapred (MRReaderMapred.java:setIncrementalConfigParams(163)) - Processing split: org.apache.hadoop.mapred.split.TezGroupedSplit@3bc4d573
2016-05-17 22:18:29,804 INFO graph.Hadoop2TezStreamGraph (Hadoop2TezStreamGraph.java:buildGraph(151)) - using streamed source: Hfs["TextDelimited[['num', 'lower']]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.5.x-tez-0.6.x/build/classes/test/data/lhs.txt"]
2016-05-17 22:18:29,804 INFO input.MRInput (MRInput.java:initFromEventInternal(634)) - Initialized RecordReader from event
2016-05-17 22:18:29,804 INFO input.MRInput (MRInput.java:processSplitEvent(586)) - Notifying on RecordReader Initialized
2016-05-17 22:18:29,805 INFO tez.FlowProcessor (FlowProcessor.java:run(121)) - sourcing from: Hfs["TextDelimited[['num', 'lower']]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.5.x-tez-0.6.x/build/classes/test/data/lhs.txt"] streamed: true, id: BB8A698F4FD141B9A6D0905721F2598E
2016-05-17 22:18:29,806 INFO tez.FlowProcessor (FlowProcessor.java:run(124)) - sinking to: GroupBy(test)[by:[{1}:'num']], id: 408CC7B95E984E308694EFFEFB566546
2016-05-17 22:18:29,806 INFO element.TezGroupGate (TezGroupGate.java:prepare(98)) - calling OrderedPartitionedKVOutput#start() on: GroupBy(test)[by:[{1}:'num']] 408CC7B95E984E308694EFFEFB566546
2016-05-17 22:18:29,811 INFO impl.ExternalSorter (ExternalSorter.java:<init>(167)) - keySerializer=cascading.tuple.hadoop.io.TuplePairSerializer@3f3c92ad; valueSerializer=cascading.tuple.hadoop.io.ValueTupleSerializer@21c1bc4a; comparator=cascading.tuple.hadoop.util.GroupingSortingComparator@3630af96; conf=cascading.tuple.hadoop.TupleSerialization,org.apache.hadoop.io.serializer.WritableSerialization,org.apache.hadoop.io.serializer.avro.AvroSpecificSerialization,org.apache.hadoop.io.serializer.avro.AvroReflectSerialization
2016-05-17 22:18:29,811 INFO impl.ExternalSorter (ExternalSorter.java:<init>(224)) - Instantiating Partitioner: [cascading.tuple.tez.util.GroupingSortingPartitioner]
2016-05-17 22:18:29,812 INFO common.TezRuntimeUtils (TezRuntimeUtils.java:instantiatePartitioner(109)) - Using partitioner class: cascading.tuple.tez.util.GroupingSortingPartitioner
2016-05-17 22:18:29,812 INFO common.TezRuntimeUtils (TezRuntimeUtils.java:instantiateCombiner(65)) - No combiner specified via tez.runtime.combiner.class. Combiner will not be used
2016-05-17 22:18:29,916 INFO dflt.DefaultSorter (DefaultSorter.java:setEquator(328)) - (EQUATOR) 0 kvi 26214396(104857584)
2016-05-17 22:18:29,916 INFO dflt.DefaultSorter (DefaultSorter.java:<init>(150)) - tez.runtime.io.sort.mb: 100
2016-05-17 22:18:29,916 INFO dflt.DefaultSorter (DefaultSorter.java:<init>(151)) - soft limit at 83886080
2016-05-17 22:18:29,916 INFO dflt.DefaultSorter (DefaultSorter.java:<init>(152)) - bufstart = 0; bufvoid = 104857600
2016-05-17 22:18:29,917 INFO dflt.DefaultSorter (DefaultSorter.java:<init>(153)) - kvstart = 26214396; length = 6553600
2016-05-17 22:18:29,917 INFO element.TezSourceStage (TezSourceStage.java:prepare(63)) - calling MRInput#start() on: Hfs["TextDelimited[['num', 'lower']]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.5.x-tez-0.6.x/build/classes/test/data/lhs.txt"]
2016-05-17 22:18:29,918 INFO input.MRInput (MRInput.java:start(450)) - MRInput setup to received events1
2016-05-17 22:18:29,918 INFO tez.FlowProcessor (FlowProcessor.java:waitForInputsReady(202)) - flow node id: A66EEB93ECE5442F8F1D1249181D9A6C, all 1 inputs ready in: 00:00:00.000
2016-05-17 22:18:29,922 INFO task.TezTaskRunner (TezTaskRunner.java:run(177)) - Closing task, taskAttemptId=attempt_1463523509443_0001_1_00_000000_0
2016-05-17 22:18:29,922 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: A66EEB93ECE5442F8F1D1249181D9A6C, mem on close (mb), free: 136, total: 382, max: 672
2016-05-17 22:18:29,922 INFO dflt.DefaultSorter (DefaultSorter.java:flush(589)) - Starting flush of map output
2016-05-17 22:18:29,923 INFO dflt.DefaultSorter (DefaultSorter.java:flush(607)) - Sorting & Spilling map output
2016-05-17 22:18:29,923 INFO dflt.DefaultSorter (DefaultSorter.java:flush(608)) - bufstart = 0; bufend = 286; bufvoid = 104857600
2016-05-17 22:18:29,923 INFO dflt.DefaultSorter (DefaultSorter.java:flush(610)) - kvstart = 26214396(104857584); kvend = 26214348(104857392); length = 49/6553600
2016-05-17 22:18:29,927 INFO dflt.DefaultSorter (DefaultSorter.java:spill(829)) - Finished spill 0
2016-05-17 22:18:29 Completed running task attempt: attempt_1463523509443_0001_1_00_000000_0
2016-05-17 22:18:29,932 INFO task.TezTaskRunner (TezTaskRunner.java:run(181)) - Task completed, taskAttemptId=attempt_1463523509443_0001_1_00_000000_0, fatalErrorOccurred=false
2016-05-17 22:18:29,934 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:cleanup(712)) - Final Counters : Counters: 27 [[File System Counters FILE_BYTES_READ=110, FILE_BYTES_WRITTEN=354, FILE_READ_OPS=0, FILE_LARGE_READ_OPS=0, FILE_WRITE_OPS=0][org.apache.tez.common.counters.TaskCounter SPILLED_RECORDS=13, GC_TIME_MILLIS=0, CPU_MILLISECONDS=0, PHYSICAL_MEMORY_BYTES=0, VIRTUAL_MEMORY_BYTES=0, COMMITTED_HEAP_BYTES=400556032, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=13, OUTPUT_BYTES=286, OUTPUT_BYTES_WITH_OVERHEAD=318, OUTPUT_BYTES_PHYSICAL=322, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=0, ADDITIONAL_SPILL_COUNT=0][cascading.flow.SliceCounters Process_Begin_Time=1463523509918, Process_Duration=4, Process_End_Time=1463523509922, Read_Duration=1, Tuples_Read=13, Tuples_Written=13, Write_Duration=1][cascading.flow.StepCounters Tuples_Read=13]]
2016-05-17 22:18:29,934 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1463523509443_0001_1][Event:VERTEX_DATA_MOVEMENT_EVENTS_GENERATED]: vertexId=vertex_1463523509443_0001_1_00, eventCount=1
2016-05-17 22:18:29,935 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event TASK_ATTEMPT_FINISHED to history file
2016-05-17 22:18:29,935 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(164)) - Redirecting log file based on addend:
2016-05-17 22:18:29,936 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(179)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-17 22:18:29,936 INFO task.ContainerReporter (ContainerReporter.java:call(55)) - Attempting to fetch new task for container container_1463523509443_0001_00_000001
2016-05-17 22:18:29,938 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1463523509443_0001_1][Event:TASK_ATTEMPT_FINISHED]: vertexName=A66EEB93ECE5442F8F1D1249181D9A6C, taskAttemptId=attempt_1463523509443_0001_1_00_000000_0, startTime=1463523509762, finishTime=1463523509935, timeTaken=173, status=SUCCEEDED, errorEnum=, diagnostics=, counters=Counters: 28, org.apache.tez.common.counters.DAGCounter, RACK_LOCAL_TASKS=1, File System Counters, FILE_BYTES_READ=110, FILE_BYTES_WRITTEN=354, FILE_READ_OPS=0, FILE_LARGE_READ_OPS=0, FILE_WRITE_OPS=0, org.apache.tez.common.counters.TaskCounter, SPILLED_RECORDS=13, GC_TIME_MILLIS=0, CPU_MILLISECONDS=0, PHYSICAL_MEMORY_BYTES=0, VIRTUAL_MEMORY_BYTES=0, COMMITTED_HEAP_BYTES=400556032, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=13, OUTPUT_BYTES=286, OUTPUT_BYTES_WITH_OVERHEAD=318, OUTPUT_BYTES_PHYSICAL=322, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=0, ADDITIONAL_SPILL_COUNT=0, cascading.flow.SliceCounters, Process_Begin_Time=1463523509918, Process_Duration=4, Process_End_Time=1463523509922, Read_Duration=1, Tuples_Read=13, Tuples_Written=13, Write_Duration=1, cascading.flow.StepCounters, Tuples_Read=13
2016-05-17 22:18:29,938 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:handle(692)) - attempt_1463523509443_0001_1_00_000000_0 TaskAttempt Transitioned from RUNNING to SUCCEEDED due to event TA_DONE
2016-05-17 22:18:29,938 INFO rm.TaskSchedulerEventHandler (TaskSchedulerEventHandler.java:handleEvent(149)) - Processing the event EventType: S_TA_ENDED
2016-05-17 22:18:29,938 INFO impl.TaskImpl (TaskImpl.java:transition(1052)) - Task succeeded with attempt attempt_1463523509443_0001_1_00_000000_0
2016-05-17 22:18:29,940 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1463523509443_0001_1][Event:TASK_FINISHED]: vertexName=A66EEB93ECE5442F8F1D1249181D9A6C, taskId=task_1463523509443_0001_1_00_000000, startTime=1463523509762, finishTime=1463523509939, timeTaken=177, status=SUCCEEDED, successfulAttemptID=attempt_1463523509443_0001_1_00_000000_0, diagnostics=, counters=Counters: 28, org.apache.tez.common.counters.DAGCounter, RACK_LOCAL_TASKS=1, File System Counters, FILE_BYTES_READ=110, FILE_BYTES_WRITTEN=354, FILE_READ_OPS=0, FILE_LARGE_READ_OPS=0, FILE_WRITE_OPS=0, org.apache.tez.common.counters.TaskCounter, SPILLED_RECORDS=13, GC_TIME_MILLIS=0, CPU_MILLISECONDS=0, PHYSICAL_MEMORY_BYTES=0, VIRTUAL_MEMORY_BYTES=0, COMMITTED_HEAP_BYTES=400556032, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=13, OUTPUT_BYTES=286, OUTPUT_BYTES_WITH_OVERHEAD=318, OUTPUT_BYTES_PHYSICAL=322, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=0, ADDITIONAL_SPILL_COUNT=0, cascading.flow.SliceCounters, Process_Begin_Time=1463523509918, Process_Duration=4, Process_End_Time=1463523509922, Read_Duration=1, Tuples_Read=13, Tuples_Written=13, Write_Duration=1, cascading.flow.StepCounters, Tuples_Read=13
2016-05-17 22:18:29,940 INFO impl.TaskImpl (TaskImpl.java:handle(862)) - task_1463523509443_0001_1_00_000000 Task Transitioned from RUNNING to SUCCEEDED due to event T_ATTEMPT_SUCCEEDED
2016-05-17 22:18:29,940 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event TASK_FINISHED to history file
2016-05-17 22:18:29,941 INFO impl.VertexImpl (VertexImpl.java:transition(3536)) - Num completed Tasks for vertex_1463523509443_0001_1_00 [A66EEB93ECE5442F8F1D1249181D9A6C] : 1
2016-05-17 22:18:29,941 INFO impl.VertexImpl (VertexImpl.java:checkVertexForCompletion(1674)) - Vertex succeeded: vertex_1463523509443_0001_1_00 [A66EEB93ECE5442F8F1D1249181D9A6C]
2016-05-17 22:18:29,943 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event VERTEX_FINISHED to history file
2016-05-17 22:18:29,944 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1463523509443_0001_1][Event:VERTEX_FINISHED]: vertexName=A66EEB93ECE5442F8F1D1249181D9A6C, vertexId=vertex_1463523509443_0001_1_00, initRequestedTime=1463523509699, initedTime=1463523509738, startRequestedTime=1463523509742, startedTime=1463523509742, finishTime=1463523509942, timeTaken=200, status=SUCCEEDED, diagnostics=, counters=Counters: 28, org.apache.tez.common.counters.DAGCounter, RACK_LOCAL_TASKS=1, File System Counters, FILE_BYTES_READ=110, FILE_BYTES_WRITTEN=354, FILE_READ_OPS=0, FILE_LARGE_READ_OPS=0, FILE_WRITE_OPS=0, org.apache.tez.common.counters.TaskCounter, SPILLED_RECORDS=13, GC_TIME_MILLIS=0, CPU_MILLISECONDS=0, PHYSICAL_MEMORY_BYTES=0, VIRTUAL_MEMORY_BYTES=0, COMMITTED_HEAP_BYTES=400556032, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=13, OUTPUT_BYTES=286, OUTPUT_BYTES_WITH_OVERHEAD=318, OUTPUT_BYTES_PHYSICAL=322, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=0, ADDITIONAL_SPILL_COUNT=0, cascading.flow.SliceCounters, Process_Begin_Time=1463523509918, Process_Duration=4, Process_End_Time=1463523509922, Read_Duration=1, Tuples_Read=13, Tuples_Written=13, Write_Duration=1, cascading.flow.StepCounters, Tuples_Read=13, vertexStats=firstTaskStartTime=1463523509762, firstTasksToStart=[ task_1463523509443_0001_1_00_000000 ], lastTaskFinishTime=1463523509935, lastTasksToFinish=[ task_1463523509443_0001_1_00_000000 ], minTaskDuration=173, maxTaskDuration=173, avgTaskDuration=173.0, numSuccessfulTasks=1, shortestDurationTasks=[ task_1463523509443_0001_1_00_000000 ], longestDurationTasks=[ task_1463523509443_0001_1_00_000000 ], vertexTaskStats={numFailedTasks=0, numSucceededTasks=1, numKilledTaskAttempts=0, numKilledTasks=0, numFailedTaskAttempts=0, numCompletedTasks=1}
2016-05-17 22:18:29,944 INFO impl.VertexImpl (VertexImpl.java:handle(1564)) - vertex_1463523509443_0001_1_00 [A66EEB93ECE5442F8F1D1249181D9A6C] transitioned from RUNNING to SUCCEEDED due to event V_TASK_COMPLETED
2016-05-17 22:18:29,944 INFO container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1463523509443_0001_00_000001 transitioned from RUNNING to IDLE via event C_TA_SUCCEEDED
2016-05-17 22:18:29,945 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1463523509443_0001_1][Event:CONTAINER_STOPPED]: containerId=container_1463523509443_0001_00_000001, stoppedTime=1463523509945, exitStatus=0
2016-05-17 22:18:29,945 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event CONTAINER_STOPPED to history file
2016-05-17 22:18:29,945 INFO container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1463523509443_0001_00_000001 transitioned from IDLE to STOP_REQUESTED via event C_STOP_REQUEST
2016-05-17 22:18:29,946 INFO impl.VertexImpl (VertexImpl.java:transition(3477)) - Source task attempt completed for vertex: vertex_1463523509443_0001_1_01 [C3F9661B4D0043369F8663BCAEB33DFE] attempt: attempt_1463523509443_0001_1_00_000000_0 with state: SUCCEEDED vertexState: RUNNING
2016-05-17 22:18:29,946 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:schedulePendingTasks(608)) - All source tasks assigned. Ramping up 1 remaining tasks for vertex: C3F9661B4D0043369F8663BCAEB33DFE
2016-05-17 22:18:29,947 INFO impl.VertexImpl (VertexImpl.java:scheduleTasks(1250)) - Routing pending task events for vertex: vertex_1463523509443_0001_1_01 [C3F9661B4D0043369F8663BCAEB33DFE]
2016-05-17 22:18:29,947 INFO impl.DAGImpl (DAGImpl.java:transition(1747)) - Vertex vertex_1463523509443_0001_1_00 [A66EEB93ECE5442F8F1D1249181D9A6C] completed., numCompletedVertices=1, numSuccessfulVertices=1, numFailedVertices=0, numKilledVertices=0, numVertices=2
2016-05-17 22:18:29,947 INFO app.TaskAttemptListenerImpTezDag (TaskAttemptListenerImpTezDag.java:getTask(213)) - Container with id: container_1463523509443_0001_00_000001 is valid, but no longer registered, and will be killed
2016-05-17 22:18:29,948 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:stop(244)) - Ignoring stop request for containerId container_1463523509443_0001_00_000001. Relying on regular task shutdown for it to end
2016-05-17 22:18:29,948 INFO task.ContainerReporter (ContainerReporter.java:call(65)) - Got TaskUpdate for containerId= container_1463523509443_0001_00_000001: 11 ms after starting to poll. TaskInfo: shouldDie: true
2016-05-17 22:18:29,948 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event TASK_STARTED to history file
2016-05-17 22:18:29,948 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1463523509443_0001_1][Event:TASK_STARTED]: vertexName=C3F9661B4D0043369F8663BCAEB33DFE, taskId=task_1463523509443_0001_1_01_000000, scheduledTime=1463523509948, launchTime=1463523509948
2016-05-17 22:18:29,948 INFO task.TezChild (TezChild.java:run(207)) - ContainerTask returned shouldDie=true for container container_1463523509443_0001_00_000001, Exiting
2016-05-17 22:18:29,949 INFO task.TezChild (TezChild.java:shutdown(343)) - Shutdown invoked for container container_1463523509443_0001_00_000001
2016-05-17 22:18:29,949 INFO task.TezChild (TezChild.java:shutdown(345)) - Shutting down container container_1463523509443_0001_00_000001
2016-05-17 22:18:29,949 INFO impl.TaskImpl (TaskImpl.java:handle(862)) - task_1463523509443_0001_1_01_000000 Task Transitioned from NEW to SCHEDULED due to event T_SCHEDULE
2016-05-17 22:18:29,952 INFO impl.DAGSchedulerNaturalOrderControlled (DAGSchedulerNaturalOrderControlled.java:scheduleTask(111)) - Scheduled vertex: vertex_1463523509443_0001_1_01 [C3F9661B4D0043369F8663BCAEB33DFE]
2016-05-17 22:18:29,952 INFO container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1463523509443_0001_00_000001 transitioned from STOP_REQUESTED to STOPPING via event C_NM_STOP_SENT
2016-05-17 22:18:29,953 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(270)) - ContainerExecutionResult for: container_1463523509443_0001_00_000001 = ContainerExecutionResult{exitStatus=SUCCESS, throwable=null, errorMessage='Asked to die by the AM'}
2016-05-17 22:18:29,953 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(274)) - Container: container_1463523509443_0001_00_000001 completed successfully
2016-05-17 22:18:29,953 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:transition(1014)) - remoteTaskSpec:DAGName : [D4BA3B0E396F46B28385B75D34E9E6FB/A57EE8E6A30E41329B7BA8BAFBA4EAD2] test/(1/1) ...irstNBufferForward/firstn, VertexName: C3F9661B4D0043369F8663BCAEB33DFE, VertexParallelism: 1, TaskAttemptID:attempt_1463523509443_0001_1_01_000000_0, processorName=cascading.flow.tez.FlowProcessor, inputSpecListSize=1, outputSpecListSize=1, inputSpecList=[{{ sourceVertexName=A66EEB93ECE5442F8F1D1249181D9A6C, physicalEdgeCount=1, inputClassName=org.apache.tez.runtime.library.input.OrderedGroupedKVInput }}, ], outputSpecList=[{{ destinationVertexName=91DD59C39FB448D58BDA96BBC66DE830, physicalEdgeCount=0, outputClassName=org.apache.tez.mapreduce.output.MROutput }}, ]
2016-05-17 22:18:29,954 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:handle(692)) - attempt_1463523509443_0001_1_01_000000_0 TaskAttempt Transitioned from NEW to START_WAIT due to event TA_SCHEDULE
2016-05-17 22:18:29,954 INFO container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1463523509443_0001_00_000001 transitioned from STOPPING to COMPLETED via event C_COMPLETED
2016-05-17 22:18:29,954 INFO rm.TaskSchedulerEventHandler (TaskSchedulerEventHandler.java:handleEvent(149)) - Processing the event EventType: S_TA_LAUNCH_REQUEST
2016-05-17 22:18:29,955 INFO container.AMContainerImpl (AMContainerImpl.java:transition(521)) - Sending Launch Request for Container with id: container_1463523509443_0001_00_000002
2016-05-17 22:18:29,956 INFO container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1463523509443_0001_00_000002 transitioned from ALLOCATED to LAUNCHING via event C_LAUNCH_REQUEST
2016-05-17 22:18:29,958 INFO task.TezChild (TezChild.java:newTezChild(432)) - PID, containerIdentifier: null, container_1463523509443_0001_00_000002
2016-05-17 22:18:29,959 WARN impl.MetricsConfig (MetricsConfig.java:loadFirst(124)) - Cannot locate configuration: tried hadoop-metrics2-teztask.properties,hadoop-metrics2.properties
2016-05-17 22:18:29,961 INFO container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1463523509443_0001_00_000002 transitioned from LAUNCHING to IDLE via event C_LAUNCHED
2016-05-17 22:18:29,961 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event CONTAINER_LAUNCHED to history file
2016-05-17 22:18:29,961 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1463523509443_0001_1][Event:CONTAINER_LAUNCHED]: containerId=container_1463523509443_0001_00_000002, launchTime=1463523509961
2016-05-17 22:18:29,962 INFO task.ContainerReporter (ContainerReporter.java:call(55)) - Attempting to fetch new task for container container_1463523509443_0001_00_000002
2016-05-17 22:18:29,963 INFO container.AMContainerImpl (AMContainerImpl.java:transition(827)) - Assigned taskAttempt + [attempt_1463523509443_0001_1_01_000000_0] to container: [container_1463523509443_0001_00_000002]
2016-05-17 22:18:29,963 INFO container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1463523509443_0001_00_000002 transitioned from IDLE to RUNNING via event C_PULL_TA
2016-05-17 22:18:29,964 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:transition(1146)) - TaskAttempt: [attempt_1463523509443_0001_1_01_000000_0] started. Is using containerId: [container_1463523509443_0001_00_000002] on NM: [127.0.0.1:0]
2016-05-17 22:18:29,964 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1463523509443_0001_1][Event:TASK_ATTEMPT_STARTED]: vertexName=C3F9661B4D0043369F8663BCAEB33DFE, taskAttemptId=attempt_1463523509443_0001_1_01_000000_0, startTime=1463523509963, containerId=container_1463523509443_0001_00_000002, nodeId=127.0.0.1:0, inProgressLogs=127.0.0.1:0/node/containerlogs/container_1463523509443_0001_00_000002/teamcity, completedLogs=
2016-05-17 22:18:29,964 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event TASK_ATTEMPT_STARTED to history file
2016-05-17 22:18:29,964 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:handle(692)) - attempt_1463523509443_0001_1_01_000000_0 TaskAttempt Transitioned from START_WAIT to RUNNING due to event TA_STARTED_REMOTELY
2016-05-17 22:18:29,964 INFO impl.TaskImpl (TaskImpl.java:handle(862)) - task_1463523509443_0001_1_01_000000 Task Transitioned from SCHEDULED to RUNNING due to event T_ATTEMPT_LAUNCHED
2016-05-17 22:18:29,965 INFO app.TaskAttemptListenerImpTezDag (TaskAttemptListenerImpTezDag.java:getTask(240)) - Container with id: container_1463523509443_0001_00_000002 given task: attempt_1463523509443_0001_1_01_000000_0
2016-05-17 22:18:29,966 INFO task.ContainerReporter (ContainerReporter.java:call(65)) - Got TaskUpdate for containerId= container_1463523509443_0001_00_000002: 0 ms after starting to poll. TaskInfo: shouldDie: false, currentTaskAttemptId: attempt_1463523509443_0001_1_01_000000_0
2016-05-17 22:18:29 Starting to run new task attempt: attempt_1463523509443_0001_1_01_000000_0
2016-05-17 22:18:29,970 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(164)) - Redirecting log file based on addend: attempt_1463523509443_0001_1_01_000000_0
2016-05-17 22:18:29,970 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(179)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-17 22:18:29,970 INFO task.TezChild (TezChild.java:handleNewTaskCredentials(273)) - Refreshing UGI since Credentials have changed
2016-05-17 22:18:29,970 INFO task.TezChild (TezChild.java:handleNewTaskCredentials(276)) - Credentials : #Tokens=1, #SecretKeys=0
2016-05-17 22:18:29,971 INFO task.TezChild (TezChild.java:handleNewTaskLocalResources(302)) - Localizing additional local resources for Task : {}
2016-05-17 22:18:29,971 INFO task.TezChild (TezChild.java:handleNewTaskLocalResources(312)) - Done localizing additional resources
2016-05-17 22:18:29,971 INFO metrics.TaskCounterUpdater (TaskCounterUpdater.java:initResourceCalculatorPlugin(148)) - Using ResourceCalculatorProcessTree : [ ]
2016-05-17 22:18:29,972 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:<init>(149)) - Initializing LogicalIOProcessorRuntimeTask with TaskSpec: DAGName : [D4BA3B0E396F46B28385B75D34E9E6FB/A57EE8E6A30E41329B7BA8BAFBA4EAD2] test/(1/1) ...irstNBufferForward/firstn, VertexName: C3F9661B4D0043369F8663BCAEB33DFE, VertexParallelism: 1, TaskAttemptID:attempt_1463523509443_0001_1_01_000000_0, processorName=cascading.flow.tez.FlowProcessor, inputSpecListSize=1, outputSpecListSize=1, inputSpecList=[{{ sourceVertexName=A66EEB93ECE5442F8F1D1249181D9A6C, physicalEdgeCount=1, inputClassName=org.apache.tez.runtime.library.input.OrderedGroupedKVInput }}, ], outputSpecList=[{{ destinationVertexName=91DD59C39FB448D58BDA96BBC66DE830, physicalEdgeCount=0, outputClassName=org.apache.tez.mapreduce.output.MROutput }}, ]
2016-05-17 22:18:29,972 INFO resources.MemoryDistributor (MemoryDistributor.java:<init>(85)) - InitialMemoryDistributor (isEnabled=true) invoked with: numInputs=1, numOutputs=1, JVM.maxFree=704643072
2016-05-17 22:18:29,975 INFO task.TezTaskRunner (TezTaskRunner.java:run(172)) - Initializing task, taskAttemptId=attempt_1463523509443_0001_1_01_000000_0
2016-05-17 22:18:29,976 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(191)) - Initializing LogicalProcessorIORuntimeTask
2016-05-17 22:18:29,976 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(195)) - Creating processor, processorClassName=cascading.flow.tez.FlowProcessor
2016-05-17 22:18:29,978 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(389)) - Initializing Input using InputSpec: { sourceVertexName=A66EEB93ECE5442F8F1D1249181D9A6C, physicalEdgeCount=1, inputClassName=org.apache.tez.runtime.library.input.OrderedGroupedKVInput }
2016-05-17 22:18:29,978 INFO impl.TaskImpl (TaskImpl.java:getTaskAttemptTezEvents(481)) - TaskAttempt:attempt_1463523509443_0001_1_01_000000_0 sent events: (0-1)
2016-05-17 22:18:29,978 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initializeLogicalIOProcessor(488)) - Initializing processor, processorClassName=cascading.flow.tez.FlowProcessor
2016-05-17 22:18:29,979 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:createInput(537)) - Creating Input
2016-05-17 22:18:29,979 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(397)) - Initializing Input with src edge: A66EEB93ECE5442F8F1D1249181D9A6C
2016-05-17 22:18:29,978 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(438)) - Initializing Output using OutputSpec: { destinationVertexName=91DD59C39FB448D58BDA96BBC66DE830, physicalEdgeCount=0, outputClassName=org.apache.tez.mapreduce.output.MROutput }
2016-05-17 22:18:29,980 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:createOutput(565)) - Creating Output
2016-05-17 22:18:29,980 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(446)) - Initializing Output with dest edge: 91DD59C39FB448D58BDA96BBC66DE830
2016-05-17 22:18:29,980 INFO output.MROutput (MROutput.java:initialize(347)) - Initializing Simple Output
2016-05-17 22:18:29,981 INFO resources.MemoryDistributor$RequestorInfo (MemoryDistributor.java:<init>(233)) - Received request: 0, type: OUTPUT, componentVertexName: 91DD59C39FB448D58BDA96BBC66DE830
2016-05-17 22:18:29,984 INFO tez.FlowProcessor (FlowProcessor.java:initialize(88)) - cascading version: 3.1.0
2016-05-17 22:18:29,992 INFO orderedgrouped.MergeManager (MergeManager.java:getInitialMemoryRequirement(310)) - Initial Shuffle Memory Required: 634178752, based on INPUT_BUFFER_factor: 0.9
2016-05-17 22:18:29,992 INFO orderedgrouped.MergeManager (MergeManager.java:getInitialMemoryRequirement(320)) - Initial Memory required for final merged output: 0, using factor: 0.0
2016-05-17 22:18:29,992 INFO resources.MemoryDistributor$RequestorInfo (MemoryDistributor.java:<init>(233)) - Received request: 634178752, type: INPUT, componentVertexName: A66EEB93ECE5442F8F1D1249181D9A6C
2016-05-17 22:18:29,998 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(403)) - Initialized Input with src edge: A66EEB93ECE5442F8F1D1249181D9A6C
2016-05-17 22:18:30,011 INFO output.MROutput (MROutput.java:initialize(418)) - Initialized Simple Output, using_new_api: false
2016-05-17 22:18:30,011 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(452)) - Initialized Output with dest edge: 91DD59C39FB448D58BDA96BBC66DE830
2016-05-17 22:18:30,011 INFO tez.FlowProcessor (FlowProcessor.java:initialize(94)) - flow node id: C3F9661B4D0043369F8663BCAEB33DFE, ordinal: 1
2016-05-17 22:18:30,012 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: C3F9661B4D0043369F8663BCAEB33DFE, mem on start (mb), free: 128, total: 382, max: 672
2016-05-17 22:18:30,012 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initializeLogicalIOProcessor(491)) - Initialized processor, processorClassName=cascading.flow.tez.FlowProcessor
2016-05-17 22:18:30,012 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(220)) - Waiting for 2 initializers to finish
2016-05-17 22:18:30,014 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(220)) - Waiting for 1 initializers to finish
2016-05-17 22:18:30,014 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(233)) - All initializers finished
2016-05-17 22:18:30,015 INFO resources.MemoryDistributor (MemoryDistributor.java:makeInitialAllocations(124)) - Using Allocator class: org.apache.tez.runtime.library.resources.WeightedScalingMemoryDistributor
2016-05-17 22:18:30,015 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:getRequestTypeForClass(197)) - Falling back to RequestType.OTHER for class: org.apache.tez.mapreduce.output.MROutput
2016-05-17 22:18:30,016 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:initialProcessMemoryRequestContext(171)) - ScaleFactor: 1, for type: OTHER
2016-05-17 22:18:30,016 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:initialProcessMemoryRequestContext(171)) - ScaleFactor: 12, for type: SORTED_MERGED_INPUT
2016-05-17 22:18:30,017 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:computeReservedFraction(262)) - InitialReservationFraction=0.3, AdditionalReservationFractionForIOs=0.03, finalReserveFractionUsed=0.32999999999999996
2016-05-17 22:18:30,017 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:assignMemory(126)) - Scaling Requests. NumRequests: 2, numScaledRequests: 13, TotalRequested: 634178752, TotalRequestedScaled: 5.853957710769231E8, TotalJVMHeap: 704643072, TotalAvailable: 472110858, TotalRequested/TotalJVMHeap:0.90
2016-05-17 22:18:30,018 INFO resources.MemoryDistributor (MemoryDistributor.java:makeInitialAllocations(138)) - Informing: OUTPUT, 91DD59C39FB448D58BDA96BBC66DE830, org.apache.tez.mapreduce.output.MROutput: requested=0, allocated=0
2016-05-17 22:18:30,019 INFO resources.MemoryDistributor (MemoryDistributor.java:makeInitialAllocations(138)) - Informing: INPUT, A66EEB93ECE5442F8F1D1249181D9A6C, org.apache.tez.runtime.library.input.OrderedGroupedKVInput: requested=634178752, allocated=472110858
2016-05-17 22:18:30,019 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(255)) - Starting Inputs/Outputs
2016-05-17 22:18:30,019 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(268)) - Input: A66EEB93ECE5442F8F1D1249181D9A6C being auto started by the framework. Subsequent instances will not be auto-started
2016-05-17 22:18:30,019 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(419)) - Starting Input with src edge: A66EEB93ECE5442F8F1D1249181D9A6C
2016-05-17 22:18:30,021 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(289)) - Num IOs determined for AutoStart: 1
2016-05-17 22:18:30,021 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(291)) - Waiting for 1 IOs to start
2016-05-17 22:18:30,022 INFO common.TezRuntimeUtils (TezRuntimeUtils.java:instantiateCombiner(65)) - No combiner specified via tez.runtime.combiner.class. Combiner will not be used
2016-05-17 22:18:30,023 INFO orderedgrouped.Shuffle (Shuffle.java:<init>(182)) - Shuffle assigned with 1 inputs, codec: NoneifileReadAhead: true
2016-05-17 22:18:30,023 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:<init>(158)) - ShuffleScheduler running for sourceVertex: A66EEB93ECE5442F8F1D1249181D9A6C with configuration: maxFetchFailuresBeforeReporting=5, reportReadErrorImmediately=true, maxFailedUniqueFetches=1, abortFailureLimit=30, maxMapRuntime=0, maxTaskOutputAtOnce=20
2016-05-17 22:18:30,023 INFO orderedgrouped.MergeManager (MergeManager.java:<init>(221)) - InitialRequest: ShuffleMem=634178752, postMergeMem=0, RuntimeTotalAvailable=472110858. Updated to: ShuffleMem=472110858, postMergeMem: 0
2016-05-17 22:18:30,024 INFO orderedgrouped.MergeManager (MergeManager.java:<init>(252)) - MergerManager: memoryLimit=472110858, maxSingleShuffleLimit=118027712, mergeThreshold=424899744, ioSortFactor=100, memToMemMergeOutputsThreshold=100
2016-05-17 22:18:30,024 INFO orderedgrouped.Shuffle (Shuffle.java:<init>(233)) - Num fetchers being started: 1
2016-05-17 22:18:30,026 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(421)) - Started Input with src edge: A66EEB93ECE5442F8F1D1249181D9A6C
2016-05-17 22:18:30,027 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(304)) - AutoStartComplete
2016-05-17 22:18:30,028 INFO task.TezTaskRunner (TezTaskRunner.java:run(175)) - Running task, taskAttemptId=attempt_1463523509443_0001_1_01_000000_0
2016-05-17 22:18:30,031 INFO graph.Hadoop2TezStreamGraph (Hadoop2TezStreamGraph.java:buildGraph(151)) - using streamed source: GroupBy(test)[by:[{1}:'num']]
2016-05-17 22:18:30,031 INFO orderedgrouped.ShuffleInputEventHandlerOrderedGrouped (ShuffleInputEventHandlerOrderedGrouped.java:processDataMovementEvent(84)) - DME srcIdx: 0, targetIdx: 0, attemptNum: 0, payload: [host: ip-10-123-148-208, port: 0, pathComponent: attempt_1463523509443_0001_1_00_000000_0_10008, runDuration: 158199, ]
2016-05-17 22:18:30,032 INFO tez.FlowProcessor (FlowProcessor.java:run(121)) - sourcing from: GroupBy(test)[by:[{1}:'num']] streamed: true, id: 408CC7B95E984E308694EFFEFB566546
2016-05-17 22:18:30,032 INFO tez.FlowProcessor (FlowProcessor.java:run(124)) - sinking to: Hfs["TextDelimited[['num', 'lower']]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.5.x-tez-0.6.x/build/test/output/hadoop2-tez/bufferpipesplatform/testFirstNBufferForward/firstn"], id: 91DD59C39FB448D58BDA96BBC66DE830
2016-05-17 22:18:30,033 INFO element.TezSinkStage (TezSinkStage.java:prepare(57)) - calling MROutput#start() on: Hfs["TextDelimited[['num', 'lower']]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.5.x-tez-0.6.x/build/test/output/hadoop2-tez/bufferpipesplatform/testFirstNBufferForward/firstn"]
2016-05-17 22:18:30,033 INFO element.TezGroupGate (TezGroupGate.java:prepare(90)) - calling OrderedGroupedKVInput#start() on: GroupBy(test)[by:[{1}:'num']] 408CC7B95E984E308694EFFEFB566546, for 1 inputs
2016-05-17 22:18:30,034 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:logIndividualFetchComplete(250)) - Completed fetch for attempt: InputAttemptIdentifier [inputIdentifier=InputIdentifier [inputIndex=0], attemptNumber=0, pathComponent=attempt_1463523509443_0001_1_00_000000_0_10008] to DISK_DIRECT, CompressedSize=322, DecompressedSize=318,EndTime=1463523510033, TimeTaken=1, Rate=0.31 MB/s
2016-05-17 22:18:30,035 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:copySucceeded(213)) - All inputs fetched for input vertex : A66EEB93ECE5442F8F1D1249181D9A6C
2016-05-17 22:18:30,035 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:logProgress(263)) - copy(1 of 1. Transfer rate (CumulativeDataFetched/TimeSinceInputStarted)) 0.00 MB/s)
2016-05-17 22:18:30,035 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:freeHost(529)) - ip-10-123-148-208:0 freed by fetcher [A66EEB93ECE5442F8F1D1249181D9A6C] #2 in 3ms
2016-05-17 22:18:30,035 INFO orderedgrouped.Shuffle (Shuffle.java:cleanupFetchers(394)) - Shutdown..fetcher [A66EEB93ECE5442F8F1D1249181D9A6C] #2, status:false, isInterrupted:false
2016-05-17 22:18:30,036 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(854)) - finalMerge called with 0 in-memory map-outputs and 1 on-disk map-outputs
2016-05-17 22:18:30,037 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(951)) - Merging 1 files, 322 bytes from disk
2016-05-17 22:18:30,037 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(966)) - Merging 0 segments, 0 bytes from memory into reduce
2016-05-17 22:18:30,037 INFO impl.TezMerger (TezMerger.java:merge(560)) - Merging 1 sorted segments
2016-05-17 22:18:30,039 INFO impl.TezMerger (TezMerger.java:merge(665)) - Down to the last merge-pass, with 1 segments left of total size: 298 bytes
2016-05-17 22:18:30,040 INFO tez.FlowProcessor (FlowProcessor.java:waitForInputsReady(202)) - flow node id: C3F9661B4D0043369F8663BCAEB33DFE, all 1 inputs ready in: 00:00:00.006
2016-05-17 22:18:30,040 INFO orderedgrouped.Shuffle (Shuffle.java:call(381)) - merge complete for input vertex : A66EEB93ECE5442F8F1D1249181D9A6C
2016-05-17 22:18:30,041 INFO orderedgrouped.Shuffle (Shuffle.java:onSuccess(488)) - Shuffle Runner thread complete
2016-05-17 22:18:30,044 INFO common.ValuesIterator (ValuesIterator.java:<init>(83)) - keyDeserializer=cascading.tuple.hadoop.io.TuplePairDeserializer@698ac596; valueDeserializer=cascading.tuple.hadoop.io.ValueTupleDeserializer@15b230fc
2016-05-17 22:18:30,047 INFO output.MROutput (MROutput.java:flush(534)) - Flushing Simple Output
2016-05-17 22:18:30,048 INFO output.MROutput (MROutput.java:flush(544)) - Flushed Simple Output
2016-05-17 22:18:30,048 INFO app.TaskAttemptListenerImpTezDag (TaskAttemptListenerImpTezDag.java:canCommit(353)) - Commit go/no-go request from attempt_1463523509443_0001_1_01_000000_0
2016-05-17 22:18:30,049 INFO impl.TaskImpl (TaskImpl.java:canCommit(750)) - attempt_1463523509443_0001_1_01_000000_0 given a go for committing the task output.
2016-05-17 22:18:30,049 INFO task.TezTaskRunner (TezTaskRunner.java:run(177)) - Closing task, taskAttemptId=attempt_1463523509443_0001_1_01_000000_0
2016-05-17 22:18:30,049 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: C3F9661B4D0043369F8663BCAEB33DFE, mem on close (mb), free: 126, total: 382, max: 672
2016-05-17 22:18:30,050 INFO orderedgrouped.Shuffle (Shuffle.java:shutdown(320)) - Shutting down Shuffle for source: A66EEB93ECE5442F8F1D1249181D9A6C
2016-05-17 22:18:30,050 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(854)) - finalMerge called with 0 in-memory map-outputs and 0 on-disk map-outputs
2016-05-17 22:18:30,051 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(951)) - Merging 0 files, 0 bytes from disk
2016-05-17 22:18:30,051 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(966)) - Merging 0 segments, 0 bytes from memory into reduce
2016-05-17 22:18:30,051 INFO impl.TezMerger (TezMerger.java:merge(560)) - Merging 0 sorted segments
2016-05-17 22:18:30,051 INFO impl.TezMerger (TezMerger.java:merge(562)) - Nothing to merge. Returning an empty iterator
2016-05-17 22:18:30 Completed running task attempt: attempt_1463523509443_0001_1_01_000000_0
2016-05-17 22:18:30,053 INFO task.TezTaskRunner (TezTaskRunner.java:run(181)) - Task completed, taskAttemptId=attempt_1463523509443_0001_1_01_000000_0, fatalErrorOccurred=false
2016-05-17 22:18:30,054 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event TASK_ATTEMPT_FINISHED to history file
2016-05-17 22:18:30,055 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1463523509443_0001_1][Event:TASK_ATTEMPT_FINISHED]: vertexName=C3F9661B4D0043369F8663BCAEB33DFE, taskAttemptId=attempt_1463523509443_0001_1_01_000000_0, startTime=1463523509963, finishTime=1463523510054, timeTaken=91, status=SUCCEEDED, errorEnum=, diagnostics=, counters=Counters: 46, File System Counters, FILE_BYTES_READ=354, FILE_BYTES_WRITTEN=32, FILE_READ_OPS=0, FILE_LARGE_READ_OPS=0, FILE_WRITE_OPS=0, org.apache.tez.common.counters.TaskCounter, REDUCE_INPUT_GROUPS=12, REDUCE_INPUT_RECORDS=5, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=13, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, GC_TIME_MILLIS=0, CPU_MILLISECONDS=0, PHYSICAL_MEMORY_BYTES=0, VIRTUAL_MEMORY_BYTES=0, COMMITTED_HEAP_BYTES=400556032, OUTPUT_RECORDS=5, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=322, SHUFFLE_BYTES=322, SHUFFLE_BYTES_DECOMPRESSED=318, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=322, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=12, MERGE_PHASE_TIME=17, FIRST_EVENT_RECEIVED=9, LAST_EVENT_RECEIVED=9, Shuffle Errors, BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0, cascading.flow.SliceCounters, Process_Begin_Time=1463523510040, Process_Duration=9, Process_End_Time=1463523510049, Read_Duration=0, Tuples_Read=5, Tuples_Written=5, Write_Duration=0, cascading.flow.StepCounters, Tuples_Written=5
2016-05-17 22:18:30,056 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:handle(692)) - attempt_1463523509443_0001_1_01_000000_0 TaskAttempt Transitioned from RUNNING to SUCCEEDED due to event TA_DONE
2016-05-17 22:18:30,056 INFO impl.TaskImpl (TaskImpl.java:transition(1052)) - Task succeeded with attempt attempt_1463523509443_0001_1_01_000000_0
2016-05-17 22:18:30,057 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event TASK_FINISHED to history file
2016-05-17 22:18:30,057 INFO rm.TaskSchedulerEventHandler (TaskSchedulerEventHandler.java:handleEvent(149)) - Processing the event EventType: S_TA_ENDED
2016-05-17 22:18:30,060 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1463523509443_0001_1][Event:TASK_FINISHED]: vertexName=C3F9661B4D0043369F8663BCAEB33DFE, taskId=task_1463523509443_0001_1_01_000000, startTime=1463523509963, finishTime=1463523510056, timeTaken=93, status=SUCCEEDED, successfulAttemptID=attempt_1463523509443_0001_1_01_000000_0, diagnostics=, counters=Counters: 46, File System Counters, FILE_BYTES_READ=354, FILE_BYTES_WRITTEN=32, FILE_READ_OPS=0, FILE_LARGE_READ_OPS=0, FILE_WRITE_OPS=0, org.apache.tez.common.counters.TaskCounter, REDUCE_INPUT_GROUPS=12, REDUCE_INPUT_RECORDS=5, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=13, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, GC_TIME_MILLIS=0, CPU_MILLISECONDS=0, PHYSICAL_MEMORY_BYTES=0, VIRTUAL_MEMORY_BYTES=0, COMMITTED_HEAP_BYTES=400556032, OUTPUT_RECORDS=5, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=322, SHUFFLE_BYTES=322, SHUFFLE_BYTES_DECOMPRESSED=318, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=322, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=12, MERGE_PHASE_TIME=17, FIRST_EVENT_RECEIVED=9, LAST_EVENT_RECEIVED=9, Shuffle Errors, BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0, cascading.flow.SliceCounters, Process_Begin_Time=1463523510040, Process_Duration=9, Process_End_Time=1463523510049, Read_Duration=0, Tuples_Read=5, Tuples_Written=5, Write_Duration=0, cascading.flow.StepCounters, Tuples_Written=5
2016-05-17 22:18:30,060 INFO impl.TaskImpl (TaskImpl.java:handle(862)) - task_1463523509443_0001_1_01_000000 Task Transitioned from RUNNING to SUCCEEDED due to event T_ATTEMPT_SUCCEEDED
2016-05-17 22:18:30,060 INFO impl.VertexImpl (VertexImpl.java:transition(3536)) - Num completed Tasks for vertex_1463523509443_0001_1_01 [C3F9661B4D0043369F8663BCAEB33DFE] : 1
2016-05-17 22:18:30,060 INFO impl.VertexImpl (VertexImpl.java:checkVertexForCompletion(1674)) - Vertex succeeded: vertex_1463523509443_0001_1_01 [C3F9661B4D0043369F8663BCAEB33DFE]
2016-05-17 22:18:30,061 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:cleanup(712)) - Final Counters : Counters: 46 [[File System Counters FILE_BYTES_READ=354, FILE_BYTES_WRITTEN=32, FILE_READ_OPS=0, FILE_LARGE_READ_OPS=0, FILE_WRITE_OPS=0][org.apache.tez.common.counters.TaskCounter REDUCE_INPUT_GROUPS=12, REDUCE_INPUT_RECORDS=5, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=13, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, GC_TIME_MILLIS=0, CPU_MILLISECONDS=0, PHYSICAL_MEMORY_BYTES=0, VIRTUAL_MEMORY_BYTES=0, COMMITTED_HEAP_BYTES=400556032, OUTPUT_RECORDS=5, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=322, SHUFFLE_BYTES=322, SHUFFLE_BYTES_DECOMPRESSED=318, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=322, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=12, MERGE_PHASE_TIME=17, FIRST_EVENT_RECEIVED=9, LAST_EVENT_RECEIVED=9][Shuffle Errors BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0][cascading.flow.SliceCounters Process_Begin_Time=1463523510040, Process_Duration=9, Process_End_Time=1463523510049, Read_Duration=0, Tuples_Read=5, Tuples_Written=5, Write_Duration=0][cascading.flow.StepCounters Tuples_Written=5]]
2016-05-17 22:18:30,061 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(164)) - Redirecting log file based on addend:
2016-05-17 22:18:30,062 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(179)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-17 22:18:30,062 INFO task.ContainerReporter (ContainerReporter.java:call(55)) - Attempting to fetch new task for container container_1463523509443_0001_00_000002
2016-05-17 22:18:30,063 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event VERTEX_FINISHED to history file
2016-05-17 22:18:30,066 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1463523509443_0001_1][Event:VERTEX_FINISHED]: vertexName=C3F9661B4D0043369F8663BCAEB33DFE, vertexId=vertex_1463523509443_0001_1_01, initRequestedTime=1463523509701, initedTime=1463523509733, startRequestedTime=1463523509748, startedTime=1463523509748, finishTime=1463523510061, timeTaken=313, status=SUCCEEDED, diagnostics=, counters=Counters: 46, File System Counters, FILE_BYTES_READ=354, FILE_BYTES_WRITTEN=32, FILE_READ_OPS=0, FILE_LARGE_READ_OPS=0, FILE_WRITE_OPS=0, org.apache.tez.common.counters.TaskCounter, REDUCE_INPUT_GROUPS=12, REDUCE_INPUT_RECORDS=5, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=13, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, GC_TIME_MILLIS=0, CPU_MILLISECONDS=0, PHYSICAL_MEMORY_BYTES=0, VIRTUAL_MEMORY_BYTES=0, COMMITTED_HEAP_BYTES=400556032, OUTPUT_RECORDS=5, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=322, SHUFFLE_BYTES=322, SHUFFLE_BYTES_DECOMPRESSED=318, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=322, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=12, MERGE_PHASE_TIME=17, FIRST_EVENT_RECEIVED=9, LAST_EVENT_RECEIVED=9, Shuffle Errors, BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0, cascading.flow.SliceCounters, Process_Begin_Time=1463523510040, Process_Duration=9, Process_End_Time=1463523510049, Read_Duration=0, Tuples_Read=5, Tuples_Written=5, Write_Duration=0, cascading.flow.StepCounters, Tuples_Written=5, vertexStats=firstTaskStartTime=1463523509963, firstTasksToStart=[ task_1463523509443_0001_1_01_000000 ], lastTaskFinishTime=1463523510054, lastTasksToFinish=[ task_1463523509443_0001_1_01_000000 ], minTaskDuration=91, maxTaskDuration=91, avgTaskDuration=91.0, numSuccessfulTasks=1, shortestDurationTasks=[ task_1463523509443_0001_1_01_000000 ], longestDurationTasks=[ task_1463523509443_0001_1_01_000000 ], vertexTaskStats={numFailedTasks=0, numSucceededTasks=1, numKilledTaskAttempts=0, numKilledTasks=0, numFailedTaskAttempts=0, numCompletedTasks=1}
2016-05-17 22:18:30,066 INFO impl.VertexImpl (VertexImpl.java:handle(1564)) - vertex_1463523509443_0001_1_01 [C3F9661B4D0043369F8663BCAEB33DFE] transitioned from RUNNING to SUCCEEDED due to event V_TASK_COMPLETED
2016-05-17 22:18:30,066 INFO container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1463523509443_0001_00_000002 transitioned from RUNNING to IDLE via event C_TA_SUCCEEDED
2016-05-17 22:18:30,067 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event CONTAINER_STOPPED to history file
2016-05-17 22:18:30,068 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1463523509443_0001_1][Event:CONTAINER_STOPPED]: containerId=container_1463523509443_0001_00_000002, stoppedTime=1463523510067, exitStatus=0
2016-05-17 22:18:30,068 INFO container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1463523509443_0001_00_000002 transitioned from IDLE to STOP_REQUESTED via event C_STOP_REQUEST
2016-05-17 22:18:30,069 INFO impl.DAGImpl (DAGImpl.java:transition(1747)) - Vertex vertex_1463523509443_0001_1_01 [C3F9661B4D0043369F8663BCAEB33DFE] completed., numCompletedVertices=2, numSuccessfulVertices=2, numFailedVertices=0, numKilledVertices=0, numVertices=2
2016-05-17 22:18:30,069 INFO impl.DAGImpl (DAGImpl.java:commitOrAbortOutputs(831)) - Calling DAG commit/abort for dag: dag_1463523509443_0001_1
2016-05-17 22:18:30,069 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1463523509443_0001_1][Event:DAG_COMMIT_STARTED]: dagID=dag_1463523509443_0001_1
2016-05-17 22:18:30,069 INFO impl.DAGImpl (DAGImpl.java:commitOrAbortOutputs(892)) - Committing output: 91DD59C39FB448D58BDA96BBC66DE830 for vertex: vertex_1463523509443_0001_1_01 [C3F9661B4D0043369F8663BCAEB33DFE]
2016-05-17 22:18:30,073 INFO app.TaskAttemptListenerImpTezDag (TaskAttemptListenerImpTezDag.java:getTask(213)) - Container with id: container_1463523509443_0001_00_000002 is valid, but no longer registered, and will be killed
2016-05-17 22:18:30,073 INFO task.ContainerReporter (ContainerReporter.java:call(65)) - Got TaskUpdate for containerId= container_1463523509443_0001_00_000002: 10 ms after starting to poll. TaskInfo: shouldDie: true
2016-05-17 22:18:30,074 INFO task.TezChild (TezChild.java:run(207)) - ContainerTask returned shouldDie=true for container container_1463523509443_0001_00_000002, Exiting
2016-05-17 22:18:30,074 INFO task.TezChild (TezChild.java:shutdown(343)) - Shutdown invoked for container container_1463523509443_0001_00_000002
2016-05-17 22:18:30,074 INFO task.TezChild (TezChild.java:shutdown(345)) - Shutting down container container_1463523509443_0001_00_000002
2016-05-17 22:18:30,075 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(270)) - ContainerExecutionResult for: container_1463523509443_0001_00_000002 = ContainerExecutionResult{exitStatus=SUCCESS, throwable=null, errorMessage='Asked to die by the AM'}
2016-05-17 22:18:30,075 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(274)) - Container: container_1463523509443_0001_00_000002 completed successfully
2016-05-17 22:18:30,080 INFO impl.DAGImpl (DAGImpl.java:commitOrAbortOutputs(871)) - No output committers for vertex: vertex_1463523509443_0001_1_00 [A66EEB93ECE5442F8F1D1249181D9A6C]
2016-05-17 22:18:30,082 INFO recovery.RecoveryService (RecoveryService.java:handle(306)) - DAG completed, dagId=dag_1463523509443_0001_1, queueSize=0
2016-05-17 22:18:30,082 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event DAG_FINISHED to history file
2016-05-17 22:18:30,084 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1463523509443_0001_1][Event:DAG_FINISHED]: dagId=dag_1463523509443_0001_1, startTime=1463523509698, finishTime=1463523510080, timeTaken=382, status=SUCCEEDED, diagnostics=, counters=Counters: 55, org.apache.tez.common.counters.DAGCounter, NUM_SUCCEEDED_TASKS=2, TOTAL_LAUNCHED_TASKS=2, RACK_LOCAL_TASKS=1, File System Counters, FILE_BYTES_READ=464, FILE_BYTES_WRITTEN=386, FILE_READ_OPS=0, FILE_LARGE_READ_OPS=0, FILE_WRITE_OPS=0, org.apache.tez.common.counters.TaskCounter, REDUCE_INPUT_GROUPS=12, REDUCE_INPUT_RECORDS=5, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=26, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, GC_TIME_MILLIS=0, CPU_MILLISECONDS=0, PHYSICAL_MEMORY_BYTES=0, VIRTUAL_MEMORY_BYTES=0, COMMITTED_HEAP_BYTES=801112064, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=18, OUTPUT_BYTES=286, OUTPUT_BYTES_WITH_OVERHEAD=318, OUTPUT_BYTES_PHYSICAL=322, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=322, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_BYTES=322, SHUFFLE_BYTES_DECOMPRESSED=318, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=322, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=12, MERGE_PHASE_TIME=17, FIRST_EVENT_RECEIVED=9, LAST_EVENT_RECEIVED=9, Shuffle Errors, BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0, cascading.flow.SliceCounters, Process_Begin_Time=2927047019958, Process_Duration=13, Process_End_Time=2927047019971, Read_Duration=1, Tuples_Read=18, Tuples_Written=18, Write_Duration=1, cascading.flow.StepCounters, Tuples_Read=13, Tuples_Written=5
2016-05-17 22:18:30,085 INFO impl.DAGImpl (DAGImpl.java:finished(1180)) - DAG: dag_1463523509443_0001_1 finished with state: SUCCEEDED
2016-05-17 22:18:30,085 INFO impl.DAGImpl (DAGImpl.java:handle(973)) - dag_1463523509443_0001_1 transitioned from RUNNING to SUCCEEDED due to event DAG_VERTEX_COMPLETED
2016-05-17 22:18:30,085 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:stop(242)) - Ignoring stop request for containerId: container_1463523509443_0001_00_000002
2016-05-17 22:18:30,085 INFO container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1463523509443_0001_00_000002 transitioned from STOP_REQUESTED to COMPLETED via event C_COMPLETED
2016-05-17 22:18:30 Completed Dag: dag_1463523509443_0001_1
2016-05-17 22:18:30,086 INFO app.DAGAppMaster (DAGAppMaster.java:handle(579)) - DAG completed, dagId=dag_1463523509443_0001_1, dagState=SUCCEEDED
2016-05-17 22:18:30,086 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(164)) - Redirecting log file based on addend: dag_1463523509443_0001_1_post
2016-05-17 22:18:30,086 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(179)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-17 22:18:30,086 INFO app.DAGAppMaster (DAGAppMaster.java:handle(625)) - Waiting for next DAG to be submitted.
2016-05-17 22:18:30,093 INFO client.TezClient (TezClient.java:stop(495)) - Shutting down Tez Session, sessionName=(1/1) ...irstNBufferForward/firstn, applicationId=application_1463523509443_0001
2016-05-17 22:18:30,095 INFO client.DAGClientHandler (DAGClientHandler.java:shutdownAM(117)) - Received message to shutdown AM
2016-05-17 22:18:30,095 INFO rm.TaskSchedulerEventHandler (TaskSchedulerEventHandler.java:setShouldUnregisterFlag(580)) - TaskScheduler notified that it should unregister from RM
2016-05-17 22:18:30,095 INFO app.DAGAppMaster (DAGAppMaster.java:shutdownTezAM(1088)) - No current running DAG, shutting down the AM
2016-05-17 22:18:30,096 INFO app.DAGAppMaster (DAGAppMaster.java:shutdown(692)) - DAGAppMasterShutdownHandler invoked
2016-05-17 22:18:30,096 INFO app.DAGAppMaster (DAGAppMaster.java:shutdown(701)) - Handling DAGAppMaster shutdown
2016-05-17 22:18:30,096 INFO app.DAGAppMaster (DAGAppMaster.java:run(721)) - Sleeping for 5 seconds before shutting down
2016-05-17 22:18:30,097 INFO history.HistoryEventHandler (HistoryEventHandler.java:serviceStop(79)) - Stopping HistoryEventHandler
2016-05-17 22:18:30,097 INFO recovery.RecoveryService (RecoveryService.java:serviceStop(189)) - Stopping RecoveryService
2016-05-17 22:18:30,097 INFO recovery.RecoveryService (RecoveryService.java:serviceStop(192)) - Handle the remaining events in queue, queue size=0
2016-05-17 22:18:30,098 INFO recovery.RecoveryService (RecoveryService.java:run(166)) - EventQueue take interrupted. Returning
2016-05-17 22:18:30,098 INFO recovery.RecoveryService (RecoveryService.java:serviceStop(214)) - Closing Summary Stream
2016-05-17 22:18:30,099 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:serviceStop(118)) - Stopping SimpleHistoryLoggingService, eventQueueBacklog=0
2016-05-17 22:18:30,099 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:run(105)) - EventQueue take interrupted. Returning
2016-05-17 22:18:30,106 INFO app.DAGAppMaster (DAGAppMaster.java:run(1773)) - Completed deletion of tez scratch data dir, path=file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.5.x-tez-0.6.x/build/test/tmp/.staging/D4BA3B0E396F46B28385B75D34E9E6FB/A57EE8E6A30E41329B7BA8BAFBA4EAD2/.tez/application_1463523509443_0001
2016-05-17 22:18:30,107 INFO util.Hadoop18TapUtil (Hadoop18TapUtil.java:cleanTempPath(226)) - deleting temp path /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.5.x-tez-0.6.x/build/test/output/hadoop2-tez/bufferpipesplatform/testFirstNBufferForward/firstn/_temporary
2016-05-17 22:18:30,108 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [test] completed in: 00:00.698
2016-05-17 22:18:30,223 INFO cascading.PlatformTestCase (PlatformTestCase.java:tearDown(189)) - copying to local /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.5.x-tez-0.6.x/build/test/output//hadoop2-tez/bufferpipesplatform/testFirstNBufferForward/firstn
2016-05-17 22:18:30,252 INFO util.Util (Util.java:findMainClass(1374)) - using default application jar, may cause class not found exceptions on the cluster
2016-05-17 22:18:30,252 INFO planner.Hadoop2TezPlanner (Hadoop2TezPlanner.java:initialize(151)) - using application jar: /opt/concurrent/home/teamcity/.gradle/caches/modules-2/files-2.1/cascading/cascading-hadoop2-tez/3.1.0/226af02e24d5195c60e4ebe911d0c7c9780f59f0/cascading-hadoop2-tez-3.1.0.jar
2016-05-17 22:18:30,321 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [test] executed rule registry: NoHashJoinHadoop2TezRuleRegistry, completed as: SUCCESS, in: 00:00.034
2016-05-17 22:18:30,341 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [test] executed rule registry: HashJoinHadoop2TezRuleRegistry, completed as: SUCCESS, in: 00:00.054
2016-05-17 22:18:30,342 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [test] rule registry: NoHashJoinHadoop2TezRuleRegistry, supports assembly with steps: 1, nodes: 2
2016-05-17 22:18:30,342 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [test] rule registry: HashJoinHadoop2TezRuleRegistry, supports assembly with steps: 1, nodes: 2
2016-05-17 22:18:30,342 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [test] rule registry: NoHashJoinHadoop2TezRuleRegistry, result was selected using: 'default comparator: selects plan with fewest steps and fewest nodes'
2016-05-17 22:18:30,365 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [test] tap forcing step to tez local mode: /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.5.x-tez-0.6.x/build/classes/test/data/lhs.txt
2016-05-17 22:18:30,374 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [test] tap forcing step to tez local mode: /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.5.x-tez-0.6.x/build/test/output/hadoop2-tez/bufferpipesplatform/testSimpleBuffer/simple
2016-05-17 22:18:30,438 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [test] starting
2016-05-17 22:18:30,438 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [test] source: Hfs["TextLine[['offset', 'line' | long, String]->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.5.x-tez-0.6.x/build/classes/test/data/lhs.txt"]
2016-05-17 22:18:30,439 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [test] sink: Hfs["TextLine[['line']->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.5.x-tez-0.6.x/build/test/output/hadoop2-tez/bufferpipesplatform/testSimpleBuffer/simple"]
2016-05-17 22:18:30,439 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [test] parallel execution of steps is enabled: false
2016-05-17 22:18:30,440 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [test] executing total steps: 1
2016-05-17 22:18:30,440 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [test] allocating management threads: 1
2016-05-17 22:18:30,441 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [test] starting step: (1/1) ...m/testSimpleBuffer/simple
2016-05-17 22:18:30,441 WARN flow.Flow (BaseFlow.java:logWarn(1516)) - [test] 'yarn.timeline-service.enabled' is disabled, please enable to capture detailed metrics of completed flows, this may require starting the YARN timeline server daemon
2016-05-17 22:18:30,457 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [test] tez session mode enabled: true
2016-05-17 22:18:30,462 INFO client.TezClient (TezClient.java:<init>(153)) - Tez Client Version: [ component=tez-api, version=0.6.2, revision=c04283bf91fd96823eec37056d5c8b1a92728265, SCM-URL=scm:git:https://git-wip-us.apache.org/repos/asf/tez.git, buildTime=20150804-1436 ]
2016-05-17 22:18:30,462 INFO client.TezClient (TezClient.java:start(333)) - Session mode. Starting session.
2016-05-17 22:18:30,463 INFO client.TezClientUtils (TezClientUtils.java:setupTezJarsLocalResources(159)) - Ignoring 'tez.lib.uris' since 'tez.ignore.lib.uris' is set to true
2016-05-17 22:18:30,473 INFO client.TezClient (TezCommonUtils.java:createTezSystemStagingPath(118)) - Tez system stage directory file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.5.x-tez-0.6.x/build/test/tmp/.staging/7D2FB5646B554769B5E79B37A91DAD15/FE1AD6A61E6A4290B0EA5055EE3472AA/.tez/application_1463523510462_0001 doesn't exist and is created
2016-05-17 22:18:30,499 INFO client.LocalClient (LocalClient.java:createDAGAppMaster(328)) - DAGAppMaster thread has been created
2016-05-17 22:18:30,499 INFO client.LocalClient (LocalClient.java:startDAGAppMaster(251)) - DAGAppMaster is not created wait for 100ms...
2016-05-17 22:18:30,500 INFO client.LocalClient (LocalClient.java:run(275)) - Using staging directory: /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.5.x-tez-0.6.x/build/test/tmp/.staging/7D2FB5646B554769B5E79B37A91DAD15/FE1AD6A61E6A4290B0EA5055EE3472AA/.tez/application_1463523510462_0001
2016-05-17 22:18:30,505 INFO app.DAGAppMaster (DAGAppMaster.java:<init>(309)) - Created DAGAppMaster for application appattempt_1463523510462_0001_000000, versionInfo=[ component=tez-dag, version=0.6.2, revision=c04283bf91fd96823eec37056d5c8b1a92728265, SCM-URL=scm:git:https://git-wip-us.apache.org/repos/asf/tez.git, buildTime=20150804-1436 ]
2016-05-17 22:18:30,518 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(333)) - Comparing client version with AM version, clientVersion=0.6.2, AMVersion=0.6.2
2016-05-17 22:18:30,519 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(386)) - Adding session token to jobTokenSecretManager for application
2016-05-17 22:18:30,520 INFO node.AMNodeTracker (AMNodeTracker.java:serviceInit(77)) - blacklistDisablePercent is 33, blacklistingEnabled: false, maxTaskFailuresPerNode: 1
2016-05-17 22:18:30,521 INFO history.HistoryEventHandler (HistoryEventHandler.java:serviceInit(50)) - Initializing HistoryEventHandler
2016-05-17 22:18:30,521 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:serviceInit(66)) - Log file location for SimpleHistoryLoggingService not specified, defaulting to containerLogDir=/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.5.x-tez-0.6.x/build/test/tmp/.staging/7D2FB5646B554769B5E79B37A91DAD15/FE1AD6A61E6A4290B0EA5055EE3472AA/.tez/application_1463523510462_0001/localmode-log-dir
2016-05-17 22:18:30,521 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:serviceInit(88)) - Initializing SimpleHistoryLoggingService, logFileLocation=/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.5.x-tez-0.6.x/build/test/tmp/.staging/7D2FB5646B554769B5E79B37A91DAD15/FE1AD6A61E6A4290B0EA5055EE3472AA/.tez/application_1463523510462_0001/localmode-log-dir/history.txt.appattempt_1463523510462_0001_000000, maxErrors=10
2016-05-17 22:18:30,521 INFO recovery.RecoveryService (RecoveryService.java:serviceInit(109)) - Initializing RecoveryService
2016-05-17 22:18:30,522 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:N/A][Event:AM_LAUNCHED]: appAttemptId=appattempt_1463523510462_0001_000000, appSubmitTime=1463523510504, launchTime=1463523510505
2016-05-17 22:18:30,529 INFO history.HistoryEventHandler (HistoryEventHandler.java:serviceStart(73)) - Starting HistoryEventHandler
2016-05-17 22:18:30,529 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:serviceStart(95)) - Starting SimpleHistoryLoggingService
2016-05-17 22:18:30,531 INFO recovery.RecoveryService (RecoveryService.java:serviceStart(127)) - Starting RecoveryService
2016-05-17 22:18:30,532 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event AM_LAUNCHED to history file
2016-05-17 22:18:30,534 INFO client.DAGClientServer (DAGClientServer.java:serviceStart(91)) - Instantiated DAGClientRPCServer at ip-10-123-148-208.ec2.internal/10.123.148.208:55539
2016-05-17 22:18:30,535 WARN impl.MetricsConfig (MetricsConfig.java:loadFirst(124)) - Cannot locate configuration: tried hadoop-metrics2-dagappmaster.properties,hadoop-metrics2.properties
2016-05-17 22:18:30,538 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event AM_STARTED to history file
2016-05-17 22:18:30,538 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:N/A][Event:AM_STARTED]: appAttemptId=appattempt_1463523510462_0001_000000, startTime=1463523510538
2016-05-17 22:18:30,539 INFO app.DAGAppMaster (DAGAppMaster.java:serviceStart(1644)) - In Session mode. Waiting for DAG over RPC
2016-05-17 22:18:30,600 INFO client.LocalClient (LocalClient.java:startDAGAppMaster(236)) - DAGAppMaster state: IDLE
2016-05-17 22:18:30,600 INFO client.TezClient (TezClient.java:start(367)) - The url to track the Tez Session: N/A
2016-05-17 22:18:30,600 INFO client.TezClient (TezClient.java:submitDAGSession(405)) - Submitting dag to TezSession, sessionName=(1/1) ...m/testSimpleBuffer/simple, applicationId=application_1463523510462_0001, dagName=[7D2FB5646B554769B5E79B37A91DAD15/FE1AD6A61E6A4290B0EA5055EE3472AA] test/(1/1) ...m/testSimpleBuffer/simple
2016-05-17 22:18:30,624 INFO app.DAGAppMaster (DAGAppMaster.java:submitDAGToAppMaster(1126)) - Starting DAG submitted via RPC: [7D2FB5646B554769B5E79B37A91DAD15/FE1AD6A61E6A4290B0EA5055EE3472AA] test/(1/1) ...m/testSimpleBuffer/simple
2016-05-17 22:18:30,626 INFO app.DAGAppMaster (DAGAppMaster.java:generateDAGVizFile(870)) - Generating DAG graphviz file, dagId=dag_1463523510462_0001_1, filePath=/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.5.x-tez-0.6.x/build/test/tmp/.staging/7D2FB5646B554769B5E79B37A91DAD15/FE1AD6A61E6A4290B0EA5055EE3472AA/.tez/application_1463523510462_0001/localmode-log-dir/dag_1463523510462_0001_1.dot
2016-05-17 22:18:30,627 INFO app.DAGAppMaster (DAGAppMaster.java:writePBTextFile(887)) - Writing DAG plan to: /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.5.x-tez-0.6.x/build/test/tmp/.staging/7D2FB5646B554769B5E79B37A91DAD15/FE1AD6A61E6A4290B0EA5055EE3472AA/.tez/application_1463523510462_0001/localmode-log-dir/dag_1463523510462_0001_1-tez-dag.pb.txt
2016-05-17 22:18:30,681 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(164)) - Redirecting log file based on addend: dag_1463523510462_0001_1
2016-05-17 22:18:30,682 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(179)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-17 22:18:30,682 INFO app.DAGAppMaster (DAGAppMaster.java:startDAG(2051)) - Running DAG: [7D2FB5646B554769B5E79B37A91DAD15/FE1AD6A61E6A4290B0EA5055EE3472AA] test/(1/1) ...m/testSimpleBuffer/simple
2016-05-17 22:18:30 Running Dag: dag_1463523510462_0001_1
2016-05-17 22:18:30,698 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1463523510462_0001_1][Event:DAG_SUBMITTED]: dagID=dag_1463523510462_0001_1, submitTime=1463523510624
2016-05-17 22:18:30,698 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event DAG_SUBMITTED to history file
2016-05-17 22:18:30,698 INFO impl.VertexImpl (VertexImpl.java:setAdditionalOutputs(4033)) - setting additional outputs for vertex C2C6CAC5F511435195BF624C09A232D5
2016-05-17 22:18:30,699 INFO impl.DAGImpl (DAGImpl.java:assignDAGScheduler(1381)) - Using DAG Scheduler: org.apache.tez.dag.app.dag.impl.DAGSchedulerNaturalOrderControlled
2016-05-17 22:18:30,700 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event DAG_INITIALIZED to history file
2016-05-17 22:18:30,700 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1463523510462_0001_1][Event:DAG_INITIALIZED]: dagID=dag_1463523510462_0001_1, initTime=1463523510698
2016-05-17 22:18:30,700 INFO impl.DAGImpl (DAGImpl.java:handle(973)) - dag_1463523510462_0001_1 transitioned from NEW to INITED due to event DAG_INIT
2016-05-17 22:18:30,701 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1463523510462_0001_1][Event:DAG_STARTED]: dagID=dag_1463523510462_0001_1, startTime=1463523510700
2016-05-17 22:18:30,701 INFO impl.DAGImpl (DAGImpl.java:transition(1614)) - Added additional resources : [[]] to classpath
2016-05-17 22:18:30,701 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event DAG_STARTED to history file
2016-05-17 22:18:30,701 INFO client.TezClient (TezClient.java:submitDAGSession(478)) - Submitted dag to TezSession, sessionName=(1/1) ...m/testSimpleBuffer/simple, applicationId=application_1463523510462_0001, dagName=[7D2FB5646B554769B5E79B37A91DAD15/FE1AD6A61E6A4290B0EA5055EE3472AA] test/(1/1) ...m/testSimpleBuffer/simple
2016-05-17 22:18:30,701 INFO impl.DAGImpl (DAGImpl.java:handle(973)) - dag_1463523510462_0001_1 transitioned from INITED to RUNNING due to event DAG_START
2016-05-17 22:18:30,702 INFO impl.VertexImpl (VertexImpl.java:setupVertex(2068)) - Root Inputs exist for Vertex: E1896BB2F1BE47659D46CCB6073BE230 : {6E180F76F8EB46C0913ED9415240A864={InputName=6E180F76F8EB46C0913ED9415240A864}, {Descriptor=ClassName=org.apache.tez.mapreduce.input.MRInput, hasPayload=true}, {ControllerDescriptor=ClassName=org.apache.tez.mapreduce.common.MRInputAMSplitGenerator, hasPayload=false}}
2016-05-17 22:18:30,703 INFO impl.VertexImpl (VertexImpl.java:setupVertex(2078)) - Starting root input initializer for input: 6E180F76F8EB46C0913ED9415240A864, with class: [org.apache.tez.mapreduce.common.MRInputAMSplitGenerator]
2016-05-17 22:18:30,703 INFO impl.VertexImpl (VertexImpl.java:assignVertexManager(2191)) - Setting vertexManager to RootInputVertexManager for vertex_1463523510462_0001_1_00 [E1896BB2F1BE47659D46CCB6073BE230]
2016-05-17 22:18:30,703 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(2983)) - Num tasks is -1. Expecting VertexManager/InputInitializers/1-1 split to set #tasks for the vertex vertex_1463523510462_0001_1_00 [E1896BB2F1BE47659D46CCB6073BE230]
2016-05-17 22:18:30,703 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(2987)) - Vertex will initialize from input initializer. vertex_1463523510462_0001_1_00 [E1896BB2F1BE47659D46CCB6073BE230]
2016-05-17 22:18:30,703 INFO impl.VertexImpl (VertexImpl.java:setupInputInitializerManager(3933)) - Vertex will initialize via inputInitializers vertex_1463523510462_0001_1_00 [E1896BB2F1BE47659D46CCB6073BE230]. Starting root input initializers: 1
2016-05-17 22:18:30,705 INFO impl.VertexImpl (VertexImpl.java:handle(1564)) - vertex_1463523510462_0001_1_00 [E1896BB2F1BE47659D46CCB6073BE230] transitioned from NEW to INITIALIZING due to event V_INIT
2016-05-17 22:18:30,705 INFO impl.VertexImpl (VertexImpl.java:assignVertexManager(2203)) - Setting vertexManager to ShuffleVertexManager for vertex_1463523510462_0001_1_01 [C2C6CAC5F511435195BF624C09A232D5]
2016-05-17 22:18:30,706 INFO dag.RootInputInitializerManager (RootInputInitializerManager.java:run(242)) - Starting InputInitializer for Input: 6E180F76F8EB46C0913ED9415240A864 on vertex vertex_1463523510462_0001_1_00 [E1896BB2F1BE47659D46CCB6073BE230]
2016-05-17 22:18:30,708 INFO common.MRInputAMSplitGenerator (MRInputAMSplitGenerator.java:initialize(92)) - Emitting serialized splits: true
2016-05-17 22:18:30,709 INFO common.MRInputAMSplitGenerator (MRInputAMSplitGenerator.java:initialize(110)) - Input 6E180F76F8EB46C0913ED9415240A864 asking for 1 tasks. Headroom: 672 Task Resource: 1024 waves: 1.7
2016-05-17 22:18:30,710 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:initialize(696)) - Shuffle Vertex Manager: settings minFrac:0.25 maxFrac:0.75 auto:false desiredTaskIput:104857600 minTasks:1
2016-05-17 22:18:30,710 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(3014)) - Creating 1 tasks for vertex: vertex_1463523510462_0001_1_01 [C2C6CAC5F511435195BF624C09A232D5]
2016-05-17 22:18:30,711 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(3026)) - Directly initializing vertex: vertex_1463523510462_0001_1_01 [C2C6CAC5F511435195BF624C09A232D5]
2016-05-17 22:18:30,711 INFO common.MRInputAMSplitGenerator (MRInputAMSplitGenerator.java:initialize(121)) - Grouping input splits
2016-05-17 22:18:30,712 INFO hadoop.MRInputHelpers (MRInputHelpers.java:generateInputSplitsToMem(294)) - Generating mapred api input splits
2016-05-17 22:18:30,711 INFO impl.VertexImpl (VertexImpl.java:initializeCommitters(1915)) - Invoking committer inits for vertex, vertexId=vertex_1463523510462_0001_1_01 [C2C6CAC5F511435195BF624C09A232D5]
2016-05-17 22:18:30,712 INFO impl.VertexImpl (VertexImpl.java:initializeCommitters(1927)) - Instantiating committer for output=BA0A5A17BB2E45CC83131A41C1DEC7C7, vertexId=vertex_1463523510462_0001_1_01 [C2C6CAC5F511435195BF624C09A232D5], committerClass=org.apache.tez.mapreduce.committer.MROutputCommitter
2016-05-17 22:18:30,713 INFO impl.VertexImpl (VertexImpl.java:run(1945)) - Invoking committer init for output=BA0A5A17BB2E45CC83131A41C1DEC7C7, vertexId=vertex_1463523510462_0001_1_01 [C2C6CAC5F511435195BF624C09A232D5]
2016-05-17 22:18:30,716 INFO hadoop.MRInputHelpers (MRInputHelpers.java:generateInputSplitsToMem(300)) - NumSplits: 1, SerializedSize: 396
2016-05-17 22:18:30,715 INFO committer.MROutputCommitter (MROutputCommitter.java:getOutputCommitter(145)) - OutputCommitter for outputName=BA0A5A17BB2E45CC83131A41C1DEC7C7, vertexName=C2C6CAC5F511435195BF624C09A232D5, outputCommitterClass=org.apache.hadoop.mapred.FileOutputCommitter
2016-05-17 22:18:30,717 INFO impl.VertexImpl (VertexImpl.java:run(1949)) - Invoking committer setup for output=BA0A5A17BB2E45CC83131A41C1DEC7C7, vertexId=vertex_1463523510462_0001_1_01 [C2C6CAC5F511435195BF624C09A232D5]
2016-05-17 22:18:30,718 INFO dag.RootInputInitializerManager (RootInputInitializerManager.java:onSuccess(272)) - Succeeded InputInitializer for Input: 6E180F76F8EB46C0913ED9415240A864 on vertex vertex_1463523510462_0001_1_00 [E1896BB2F1BE47659D46CCB6073BE230]
2016-05-17 22:18:30,719 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event VERTEX_INITIALIZED to history file
2016-05-17 22:18:30,719 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1463523510462_0001_1][Event:VERTEX_INITIALIZED]: vertexName=C2C6CAC5F511435195BF624C09A232D5, vertexId=vertex_1463523510462_0001_1_01, initRequestedTime=1463523510705, initedTime=1463523510719, numTasks=1, processorName=cascading.flow.tez.FlowProcessor, additionalInputsCount=0
2016-05-17 22:18:30,719 INFO impl.VertexImpl (VertexImpl.java:handle(1564)) - vertex_1463523510462_0001_1_01 [C2C6CAC5F511435195BF624C09A232D5] transitioned from NEW to INITED due to event V_INIT
2016-05-17 22:18:30,720 INFO impl.VertexImpl (VertexImpl.java:canInitVertex(3195)) - Cannot init vertex: vertex_1463523510462_0001_1_00 [E1896BB2F1BE47659D46CCB6073BE230] numTasks: -1 numUnitializedEdges: 0 numInitializedInputs: 0 initWaitsForRootInitializers: true
2016-05-17 22:18:30,720 INFO impl.VertexImpl (VertexImpl.java:setParallelism(1374)) - Got updated RootInputsSpecs: {6E180F76F8EB46C0913ED9415240A864=forAllWorkUnits=true, update=[1]}
2016-05-17 22:18:30,720 INFO impl.VertexImpl (VertexImpl.java:setParallelism(1393)) - Vertex vertex_1463523510462_0001_1_00 [E1896BB2F1BE47659D46CCB6073BE230] parallelism set to 1
2016-05-17 22:18:30,720 INFO impl.VertexImpl (VertexImpl.java:canInitVertex(3195)) - Cannot init vertex: vertex_1463523510462_0001_1_00 [E1896BB2F1BE47659D46CCB6073BE230] numTasks: 1 numUnitializedEdges: 0 numInitializedInputs: 0 initWaitsForRootInitializers: true
2016-05-17 22:18:30,721 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1463523510462_0001_1][Event:VERTEX_DATA_MOVEMENT_EVENTS_GENERATED]: vertexId=vertex_1463523510462_0001_1_00, eventCount=1
2016-05-17 22:18:30,721 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event VERTEX_INITIALIZED to history file
2016-05-17 22:18:30,722 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1463523510462_0001_1][Event:VERTEX_INITIALIZED]: vertexName=E1896BB2F1BE47659D46CCB6073BE230, vertexId=vertex_1463523510462_0001_1_00, initRequestedTime=1463523510702, initedTime=1463523510721, numTasks=1, processorName=cascading.flow.tez.FlowProcessor, additionalInputsCount=1
2016-05-17 22:18:30,722 INFO impl.VertexImpl (VertexImpl.java:startIfPossible(3061)) - Triggering start event for vertex: vertex_1463523510462_0001_1_00 [E1896BB2F1BE47659D46CCB6073BE230] with distanceFromRoot: 0
2016-05-17 22:18:30,722 INFO impl.VertexImpl (VertexImpl.java:handle(1564)) - vertex_1463523510462_0001_1_00 [E1896BB2F1BE47659D46CCB6073BE230] transitioned from INITIALIZING to INITED due to event V_ROOT_INPUT_INITIALIZED
2016-05-17 22:18:30,723 INFO impl.ImmediateStartVertexManager (ImmediateStartVertexManager.java:scheduleTasks(99)) - Starting 1 in E1896BB2F1BE47659D46CCB6073BE230
2016-05-17 22:18:30,723 INFO impl.VertexImpl (VertexImpl.java:scheduleTasks(1250)) - Routing pending task events for vertex: vertex_1463523510462_0001_1_00 [E1896BB2F1BE47659D46CCB6073BE230]
2016-05-17 22:18:30,723 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event VERTEX_STARTED to history file
2016-05-17 22:18:30,724 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1463523510462_0001_1][Event:VERTEX_STARTED]: vertexId=vertex_1463523510462_0001_1_00, startRequestedTime=1463523510723, startedTime=1463523510723
2016-05-17 22:18:30,724 INFO impl.VertexImpl (VertexImpl.java:handle(1564)) - vertex_1463523510462_0001_1_00 [E1896BB2F1BE47659D46CCB6073BE230] transitioned from INITED to RUNNING due to event V_START
2016-05-17 22:18:30,725 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event TASK_STARTED to history file
2016-05-17 22:18:30,726 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1463523510462_0001_1][Event:TASK_STARTED]: vertexName=E1896BB2F1BE47659D46CCB6073BE230, taskId=task_1463523510462_0001_1_00_000000, scheduledTime=1463523510725, launchTime=1463523510725
2016-05-17 22:18:30,726 INFO impl.TaskImpl (TaskImpl.java:handle(862)) - task_1463523510462_0001_1_00_000000 Task Transitioned from NEW to SCHEDULED due to event T_SCHEDULE
2016-05-17 22:18:30,726 INFO impl.VertexImpl (VertexImpl.java:transition(3155)) - Source vertex started: vertex_1463523510462_0001_1_00 for vertex: vertex_1463523510462_0001_1_01 [C2C6CAC5F511435195BF624C09A232D5] numStartedSources: 1 numSources: 1
2016-05-17 22:18:30,727 INFO impl.VertexImpl (VertexImpl.java:startIfPossible(3061)) - Triggering start event for vertex: vertex_1463523510462_0001_1_01 [C2C6CAC5F511435195BF624C09A232D5] with distanceFromRoot: 1
2016-05-17 22:18:30,727 INFO impl.DAGSchedulerNaturalOrderControlled (DAGSchedulerNaturalOrderControlled.java:scheduleTask(111)) - Scheduled vertex: vertex_1463523510462_0001_1_00 [E1896BB2F1BE47659D46CCB6073BE230]
2016-05-17 22:18:30,728 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:handleVertexStateUpdate(720)) - Received configured notification : CONFIGURED for vertex: E1896BB2F1BE47659D46CCB6073BE230 in vertex: C2C6CAC5F511435195BF624C09A232D5
2016-05-17 22:18:30,729 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:onVertexStarted(355)) - OnVertexStarted vertex: C2C6CAC5F511435195BF624C09A232D5 with 1 source tasks and 1 pending tasks
2016-05-17 22:18:30,729 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [test] submitted tez dag to app master: application_1463523510462_0001, with dag id: dag_1463523510462_0001_1
2016-05-17 22:18:30,729 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event VERTEX_STARTED to history file
2016-05-17 22:18:30,729 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1463523510462_0001_1][Event:VERTEX_STARTED]: vertexId=vertex_1463523510462_0001_1_01, startRequestedTime=1463523510728, startedTime=1463523510728
2016-05-17 22:18:30,730 INFO impl.VertexImpl (VertexImpl.java:handle(1564)) - vertex_1463523510462_0001_1_01 [C2C6CAC5F511435195BF624C09A232D5] transitioned from INITED to RUNNING due to event V_START
2016-05-17 22:18:30,730 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:transition(1014)) - remoteTaskSpec:DAGName : [7D2FB5646B554769B5E79B37A91DAD15/FE1AD6A61E6A4290B0EA5055EE3472AA] test/(1/1) ...m/testSimpleBuffer/simple, VertexName: E1896BB2F1BE47659D46CCB6073BE230, VertexParallelism: 1, TaskAttemptID:attempt_1463523510462_0001_1_00_000000_0, processorName=cascading.flow.tez.FlowProcessor, inputSpecListSize=1, outputSpecListSize=1, inputSpecList=[{{ sourceVertexName=6E180F76F8EB46C0913ED9415240A864, physicalEdgeCount=1, inputClassName=org.apache.tez.mapreduce.input.MRInput }}, ], outputSpecList=[{{ destinationVertexName=C2C6CAC5F511435195BF624C09A232D5, physicalEdgeCount=1, outputClassName=org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput }}, ]
2016-05-17 22:18:30,731 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:handle(692)) - attempt_1463523510462_0001_1_00_000000_0 TaskAttempt Transitioned from NEW to START_WAIT due to event TA_SCHEDULE
2016-05-17 22:18:30,731 INFO rm.TaskSchedulerEventHandler (TaskSchedulerEventHandler.java:handleEvent(149)) - Processing the event EventType: S_TA_LAUNCH_REQUEST
2016-05-17 22:18:30,732 INFO node.AMNodeTracker (AMNodeTracker.java:nodeSeen(91)) - Adding new node: 127.0.0.1:0
2016-05-17 22:18:30,732 INFO container.AMContainerHelpers (AMContainerHelpers.java:createCommonContainerLaunchContext(112)) - Adding #1 tokens and #0 secret keys for NM use for launching container
2016-05-17 22:18:30,733 INFO container.AMContainerHelpers (AMContainerHelpers.java:createCommonContainerLaunchContext(122)) - Putting shuffle token in serviceData
2016-05-17 22:18:30,735 INFO container.AMContainerImpl (AMContainerImpl.java:transition(521)) - Sending Launch Request for Container with id: container_1463523510462_0001_00_000001
2016-05-17 22:18:30,735 INFO container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1463523510462_0001_00_000001 transitioned from ALLOCATED to LAUNCHING via event C_LAUNCH_REQUEST
2016-05-17 22:18:30,738 INFO task.TezChild (TezChild.java:newTezChild(432)) - PID, containerIdentifier: null, container_1463523510462_0001_00_000001
2016-05-17 22:18:30,738 WARN impl.MetricsSystemImpl (MetricsSystemImpl.java:init(145)) - DAGAppMaster metrics system already initialized!
2016-05-17 22:18:30,739 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event CONTAINER_LAUNCHED to history file
2016-05-17 22:18:30,739 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1463523510462_0001_1][Event:CONTAINER_LAUNCHED]: containerId=container_1463523510462_0001_00_000001, launchTime=1463523510739
2016-05-17 22:18:30,740 INFO container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1463523510462_0001_00_000001 transitioned from LAUNCHING to IDLE via event C_LAUNCHED
2016-05-17 22:18:30,741 INFO task.ContainerReporter (ContainerReporter.java:call(55)) - Attempting to fetch new task for container container_1463523510462_0001_00_000001
2016-05-17 22:18:30,743 INFO container.AMContainerImpl (AMContainerImpl.java:transition(827)) - Assigned taskAttempt + [attempt_1463523510462_0001_1_00_000000_0] to container: [container_1463523510462_0001_00_000001]
2016-05-17 22:18:30,743 INFO container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1463523510462_0001_00_000001 transitioned from IDLE to RUNNING via event C_PULL_TA
2016-05-17 22:18:30,743 INFO app.TaskAttemptListenerImpTezDag (TaskAttemptListenerImpTezDag.java:getTask(240)) - Container with id: container_1463523510462_0001_00_000001 given task: attempt_1463523510462_0001_1_00_000000_0
2016-05-17 22:18:30,743 INFO task.ContainerReporter (ContainerReporter.java:call(65)) - Got TaskUpdate for containerId= container_1463523510462_0001_00_000001: 0 ms after starting to poll. TaskInfo: shouldDie: false, currentTaskAttemptId: attempt_1463523510462_0001_1_00_000000_0
2016-05-17 22:18:30,743 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:transition(1146)) - TaskAttempt: [attempt_1463523510462_0001_1_00_000000_0] started. Is using containerId: [container_1463523510462_0001_00_000001] on NM: [127.0.0.1:0]
2016-05-17 22:18:30 Starting to run new task attempt: attempt_1463523510462_0001_1_00_000000_0
2016-05-17 22:18:30,744 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(164)) - Redirecting log file based on addend: attempt_1463523510462_0001_1_00_000000_0
2016-05-17 22:18:30,744 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(179)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-17 22:18:30,744 INFO task.TezChild (TezChild.java:handleNewTaskCredentials(273)) - Refreshing UGI since Credentials have changed
2016-05-17 22:18:30,744 INFO task.TezChild (TezChild.java:handleNewTaskCredentials(276)) - Credentials : #Tokens=1, #SecretKeys=0
2016-05-17 22:18:30,744 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event TASK_ATTEMPT_STARTED to history file
2016-05-17 22:18:30,745 INFO task.TezChild (TezChild.java:handleNewTaskLocalResources(302)) - Localizing additional local resources for Task : {}
2016-05-17 22:18:30,745 INFO task.TezChild (TezChild.java:handleNewTaskLocalResources(312)) - Done localizing additional resources
2016-05-17 22:18:30,745 INFO metrics.TaskCounterUpdater (TaskCounterUpdater.java:initResourceCalculatorPlugin(148)) - Using ResourceCalculatorProcessTree : [ ]
2016-05-17 22:18:30,745 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:<init>(149)) - Initializing LogicalIOProcessorRuntimeTask with TaskSpec: DAGName : [7D2FB5646B554769B5E79B37A91DAD15/FE1AD6A61E6A4290B0EA5055EE3472AA] test/(1/1) ...m/testSimpleBuffer/simple, VertexName: E1896BB2F1BE47659D46CCB6073BE230, VertexParallelism: 1, TaskAttemptID:attempt_1463523510462_0001_1_00_000000_0, processorName=cascading.flow.tez.FlowProcessor, inputSpecListSize=1, outputSpecListSize=1, inputSpecList=[{{ sourceVertexName=6E180F76F8EB46C0913ED9415240A864, physicalEdgeCount=1, inputClassName=org.apache.tez.mapreduce.input.MRInput }}, ], outputSpecList=[{{ destinationVertexName=C2C6CAC5F511435195BF624C09A232D5, physicalEdgeCount=1, outputClassName=org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput }}, ]
2016-05-17 22:18:30,745 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1463523510462_0001_1][Event:TASK_ATTEMPT_STARTED]: vertexName=E1896BB2F1BE47659D46CCB6073BE230, taskAttemptId=attempt_1463523510462_0001_1_00_000000_0, startTime=1463523510743, containerId=container_1463523510462_0001_00_000001, nodeId=127.0.0.1:0, inProgressLogs=127.0.0.1:0/node/containerlogs/container_1463523510462_0001_00_000001/teamcity, completedLogs=
2016-05-17 22:18:30,746 INFO resources.MemoryDistributor (MemoryDistributor.java:<init>(85)) - InitialMemoryDistributor (isEnabled=true) invoked with: numInputs=1, numOutputs=1, JVM.maxFree=704643072
2016-05-17 22:18:30,746 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:handle(692)) - attempt_1463523510462_0001_1_00_000000_0 TaskAttempt Transitioned from START_WAIT to RUNNING due to event TA_STARTED_REMOTELY
2016-05-17 22:18:30,746 INFO impl.TaskImpl (TaskImpl.java:handle(862)) - task_1463523510462_0001_1_00_000000 Task Transitioned from SCHEDULED to RUNNING due to event T_ATTEMPT_LAUNCHED
2016-05-17 22:18:30,746 INFO task.TezTaskRunner (TezTaskRunner.java:run(172)) - Initializing task, taskAttemptId=attempt_1463523510462_0001_1_00_000000_0
2016-05-17 22:18:30,747 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(191)) - Initializing LogicalProcessorIORuntimeTask
2016-05-17 22:18:30,747 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(195)) - Creating processor, processorClassName=cascading.flow.tez.FlowProcessor
2016-05-17 22:18:30,748 INFO impl.TaskImpl (TaskImpl.java:getTaskAttemptTezEvents(481)) - TaskAttempt:attempt_1463523510462_0001_1_00_000000_0 sent events: (0-1)
2016-05-17 22:18:30,748 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initializeLogicalIOProcessor(488)) - Initializing processor, processorClassName=cascading.flow.tez.FlowProcessor
2016-05-17 22:18:30,748 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(389)) - Initializing Input using InputSpec: { sourceVertexName=6E180F76F8EB46C0913ED9415240A864, physicalEdgeCount=1, inputClassName=org.apache.tez.mapreduce.input.MRInput }
2016-05-17 22:18:30,749 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:createInput(537)) - Creating Input
2016-05-17 22:18:30,749 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(397)) - Initializing Input with src edge: 6E180F76F8EB46C0913ED9415240A864
2016-05-17 22:18:30,749 INFO resources.MemoryDistributor$RequestorInfo (MemoryDistributor.java:<init>(233)) - Received request: 0, type: INPUT, componentVertexName: 6E180F76F8EB46C0913ED9415240A864
2016-05-17 22:18:30,751 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(438)) - Initializing Output using OutputSpec: { destinationVertexName=C2C6CAC5F511435195BF624C09A232D5, physicalEdgeCount=1, outputClassName=org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput }
2016-05-17 22:18:30,751 INFO tez.FlowProcessor (FlowProcessor.java:initialize(88)) - cascading version: 3.1.0
2016-05-17 22:18:30,752 INFO input.MRInput (MRInput.java:initialize(440)) - Using New mapreduce API: false, split information via event: true
2016-05-17 22:18:30,753 INFO input.MRInput (MRInput.java:initializeInternal(493)) - Initialzed MRInput: 6E180F76F8EB46C0913ED9415240A864
2016-05-17 22:18:30,754 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(403)) - Initialized Input with src edge: 6E180F76F8EB46C0913ED9415240A864
2016-05-17 22:18:30,751 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:createOutput(565)) - Creating Output
2016-05-17 22:18:30,754 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(446)) - Initializing Output with dest edge: C2C6CAC5F511435195BF624C09A232D5
2016-05-17 22:18:30,757 INFO impl.ExternalSorter (ExternalSorter.java:getInitialMemoryRequirement(293)) - Requested SortBufferSize (tez.runtime.io.sort.mb): 100
2016-05-17 22:18:30,757 INFO resources.MemoryDistributor$RequestorInfo (MemoryDistributor.java:<init>(233)) - Received request: 104857600, type: OUTPUT, componentVertexName: C2C6CAC5F511435195BF624C09A232D5
2016-05-17 22:18:30,757 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(452)) - Initialized Output with dest edge: C2C6CAC5F511435195BF624C09A232D5
2016-05-17 22:18:30,761 INFO tez.FlowProcessor (FlowProcessor.java:initialize(94)) - flow node id: E1896BB2F1BE47659D46CCB6073BE230, ordinal: 0
2016-05-17 22:18:30,761 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: E1896BB2F1BE47659D46CCB6073BE230, mem on start (mb), free: 250, total: 490, max: 672
2016-05-17 22:18:30,761 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initializeLogicalIOProcessor(491)) - Initialized processor, processorClassName=cascading.flow.tez.FlowProcessor
2016-05-17 22:18:30,761 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(220)) - Waiting for 2 initializers to finish
2016-05-17 22:18:30,761 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(220)) - Waiting for 1 initializers to finish
2016-05-17 22:18:30,761 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(233)) - All initializers finished
2016-05-17 22:18:30,761 INFO resources.MemoryDistributor (MemoryDistributor.java:makeInitialAllocations(124)) - Using Allocator class: org.apache.tez.runtime.library.resources.WeightedScalingMemoryDistributor
2016-05-17 22:18:30,762 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:getRequestTypeForClass(197)) - Falling back to RequestType.OTHER for class: org.apache.tez.mapreduce.input.MRInput
2016-05-17 22:18:30,762 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:initialProcessMemoryRequestContext(171)) - ScaleFactor: 1, for type: OTHER
2016-05-17 22:18:30,762 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:initialProcessMemoryRequestContext(171)) - ScaleFactor: 12, for type: SORTED_OUTPUT
2016-05-17 22:18:30,762 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:computeReservedFraction(262)) - InitialReservationFraction=0.3, AdditionalReservationFractionForIOs=0.03, finalReserveFractionUsed=0.32999999999999996
2016-05-17 22:18:30,762 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:assignMemory(126)) - Scaling Requests. NumRequests: 2, numScaledRequests: 13, TotalRequested: 104857600, TotalRequestedScaled: 9.679163076923077E7, TotalJVMHeap: 704643072, TotalAvailable: 472110858, TotalRequested/TotalJVMHeap:0.15
2016-05-17 22:18:30,763 INFO resources.MemoryDistributor (MemoryDistributor.java:makeInitialAllocations(138)) - Informing: INPUT, 6E180F76F8EB46C0913ED9415240A864, org.apache.tez.mapreduce.input.MRInput: requested=0, allocated=0
2016-05-17 22:18:30,763 INFO resources.MemoryDistributor (MemoryDistributor.java:makeInitialAllocations(138)) - Informing: OUTPUT, C2C6CAC5F511435195BF624C09A232D5, org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput: requested=104857600, allocated=104857600
2016-05-17 22:18:30,763 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(255)) - Starting Inputs/Outputs
2016-05-17 22:18:30,763 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(268)) - Input: 6E180F76F8EB46C0913ED9415240A864 being auto started by the framework. Subsequent instances will not be auto-started
2016-05-17 22:18:30,763 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(289)) - Num IOs determined for AutoStart: 1
2016-05-17 22:18:30,763 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(419)) - Starting Input with src edge: 6E180F76F8EB46C0913ED9415240A864
2016-05-17 22:18:30,763 INFO input.MRInput (MRInput.java:start(450)) - MRInput setup to received events1
2016-05-17 22:18:30,763 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(421)) - Started Input with src edge: 6E180F76F8EB46C0913ED9415240A864
2016-05-17 22:18:30,763 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(291)) - Waiting for 1 IOs to start
2016-05-17 22:18:30,764 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(304)) - AutoStartComplete
2016-05-17 22:18:30,764 INFO task.TezTaskRunner (TezTaskRunner.java:run(175)) - Running task, taskAttemptId=attempt_1463523510462_0001_1_00_000000_0
2016-05-17 22:18:30,764 INFO input.MRInput (MRInput.java:initFromEventInternal(619)) - Initializing RecordReader from event
2016-05-17 22:18:30,765 INFO input.MRInput (MRInput.java:initFromEventInternal(630)) - Split Details -> SplitClass: org.apache.hadoop.mapred.split.TezGroupedSplit, OldSplit: org.apache.hadoop.mapred.split.TezGroupedSplit@337e879c
2016-05-17 22:18:30,767 INFO graph.Hadoop2TezStreamGraph (Hadoop2TezStreamGraph.java:buildGraph(151)) - using streamed source: Hfs["TextLine[['offset', 'line' | long, String]->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.5.x-tez-0.6.x/build/classes/test/data/lhs.txt"]
2016-05-17 22:18:30,767 INFO lib.MRReaderMapred (MRReaderMapred.java:setIncrementalConfigParams(163)) - Processing split: org.apache.hadoop.mapred.split.TezGroupedSplit@337e879c
2016-05-17 22:18:30,768 INFO input.MRInput (MRInput.java:initFromEventInternal(634)) - Initialized RecordReader from event
2016-05-17 22:18:30,768 INFO tez.FlowProcessor (FlowProcessor.java:run(121)) - sourcing from: Hfs["TextLine[['offset', 'line' | long, String]->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.5.x-tez-0.6.x/build/classes/test/data/lhs.txt"] streamed: true, id: 6E180F76F8EB46C0913ED9415240A864
2016-05-17 22:18:30,768 INFO tez.FlowProcessor (FlowProcessor.java:run(124)) - sinking to: GroupBy(test)[by:[{1}:'num']], id: E43200532F7043DA95777CF1D2395C76
2016-05-17 22:18:30,768 INFO input.MRInput (MRInput.java:processSplitEvent(586)) - Notifying on RecordReader Initialized
2016-05-17 22:18:30,768 INFO element.TezGroupGate (TezGroupGate.java:prepare(98)) - calling OrderedPartitionedKVOutput#start() on: GroupBy(test)[by:[{1}:'num']] E43200532F7043DA95777CF1D2395C76
2016-05-17 22:18:30,772 INFO impl.ExternalSorter (ExternalSorter.java:<init>(167)) - keySerializer=cascading.tuple.hadoop.io.KeyTupleSerializer@588ac488; valueSerializer=cascading.tuple.hadoop.io.ValueTupleSerializer@7e6cea21; comparator=cascading.tuple.hadoop.util.TupleComparator@207f5d5f; conf=cascading.tuple.hadoop.TupleSerialization,org.apache.hadoop.io.serializer.WritableSerialization,org.apache.hadoop.io.serializer.avro.AvroSpecificSerialization,org.apache.hadoop.io.serializer.avro.AvroReflectSerialization
2016-05-17 22:18:30,773 INFO impl.ExternalSorter (ExternalSorter.java:<init>(224)) - Instantiating Partitioner: [cascading.tuple.tez.util.TuplePartitioner]
2016-05-17 22:18:30,773 INFO common.TezRuntimeUtils (TezRuntimeUtils.java:instantiatePartitioner(109)) - Using partitioner class: cascading.tuple.tez.util.TuplePartitioner
2016-05-17 22:18:30,773 INFO common.TezRuntimeUtils (TezRuntimeUtils.java:instantiateCombiner(65)) - No combiner specified via tez.runtime.combiner.class. Combiner will not be used
2016-05-17 22:18:30,788 INFO dflt.DefaultSorter (DefaultSorter.java:setEquator(328)) - (EQUATOR) 0 kvi 26214396(104857584)
2016-05-17 22:18:30,789 INFO dflt.DefaultSorter (DefaultSorter.java:<init>(150)) - tez.runtime.io.sort.mb: 100
2016-05-17 22:18:30,789 INFO dflt.DefaultSorter (DefaultSorter.java:<init>(151)) - soft limit at 83886080
2016-05-17 22:18:30,789 INFO dflt.DefaultSorter (DefaultSorter.java:<init>(152)) - bufstart = 0; bufvoid = 104857600
2016-05-17 22:18:30,789 INFO dflt.DefaultSorter (DefaultSorter.java:<init>(153)) - kvstart = 26214396; length = 6553600
2016-05-17 22:18:30,790 INFO element.TezSourceStage (TezSourceStage.java:prepare(63)) - calling MRInput#start() on: Hfs["TextLine[['offset', 'line' | long, String]->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.5.x-tez-0.6.x/build/classes/test/data/lhs.txt"]
2016-05-17 22:18:30,790 INFO input.MRInput (MRInput.java:start(450)) - MRInput setup to received events1
2016-05-17 22:18:30,790 INFO tez.FlowProcessor (FlowProcessor.java:waitForInputsReady(202)) - flow node id: E1896BB2F1BE47659D46CCB6073BE230, all 1 inputs ready in: 00:00:00.000
2016-05-17 22:18:30,793 INFO task.TezTaskRunner (TezTaskRunner.java:run(177)) - Closing task, taskAttemptId=attempt_1463523510462_0001_1_00_000000_0
2016-05-17 22:18:30,793 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: E1896BB2F1BE47659D46CCB6073BE230, mem on close (mb), free: 148, total: 490, max: 672
2016-05-17 22:18:30,793 INFO dflt.DefaultSorter (DefaultSorter.java:flush(589)) - Starting flush of map output
2016-05-17 22:18:30,793 INFO dflt.DefaultSorter (DefaultSorter.java:flush(607)) - Sorting & Spilling map output
2016-05-17 22:18:30,793 INFO dflt.DefaultSorter (DefaultSorter.java:flush(608)) - bufstart = 0; bufend = 195; bufvoid = 104857600
2016-05-17 22:18:30,793 INFO dflt.DefaultSorter (DefaultSorter.java:flush(610)) - kvstart = 26214396(104857584); kvend = 26214348(104857392); length = 49/6553600
2016-05-17 22:18:30,800 INFO dflt.DefaultSorter (DefaultSorter.java:spill(829)) - Finished spill 0
2016-05-17 22:18:30 Completed running task attempt: attempt_1463523510462_0001_1_00_000000_0
2016-05-17 22:18:30,805 INFO task.TezTaskRunner (TezTaskRunner.java:run(181)) - Task completed, taskAttemptId=attempt_1463523510462_0001_1_00_000000_0, fatalErrorOccurred=false
2016-05-17 22:18:30,806 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1463523510462_0001_1][Event:VERTEX_DATA_MOVEMENT_EVENTS_GENERATED]: vertexId=vertex_1463523510462_0001_1_00, eventCount=1
2016-05-17 22:18:30,806 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event TASK_ATTEMPT_FINISHED to history file
2016-05-17 22:18:30,807 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1463523510462_0001_1][Event:TASK_ATTEMPT_FINISHED]: vertexName=E1896BB2F1BE47659D46CCB6073BE230, taskAttemptId=attempt_1463523510462_0001_1_00_000000_0, startTime=1463523510743, finishTime=1463523510806, timeTaken=63, status=SUCCEEDED, errorEnum=, diagnostics=, counters=Counters: 28, org.apache.tez.common.counters.DAGCounter, RACK_LOCAL_TASKS=1, File System Counters, FILE_BYTES_READ=110, FILE_BYTES_WRITTEN=727, FILE_READ_OPS=0, FILE_LARGE_READ_OPS=0, FILE_WRITE_OPS=0, org.apache.tez.common.counters.TaskCounter, SPILLED_RECORDS=13, GC_TIME_MILLIS=0, CPU_MILLISECONDS=0, PHYSICAL_MEMORY_BYTES=0, VIRTUAL_MEMORY_BYTES=0, COMMITTED_HEAP_BYTES=513802240, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=13, OUTPUT_BYTES=195, OUTPUT_BYTES_WITH_OVERHEAD=171, OUTPUT_BYTES_PHYSICAL=175, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=0, ADDITIONAL_SPILL_COUNT=0, cascading.flow.SliceCounters, Process_Begin_Time=1463523510790, Process_Duration=3, Process_End_Time=1463523510793, Read_Duration=2, Tuples_Read=13, Tuples_Written=13, Write_Duration=0, cascading.flow.StepCounters, Tuples_Read=13
2016-05-17 22:18:30,808 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:handle(692)) - attempt_1463523510462_0001_1_00_000000_0 TaskAttempt Transitioned from RUNNING to SUCCEEDED due to event TA_DONE
2016-05-17 22:18:30,808 INFO impl.TaskImpl (TaskImpl.java:transition(1052)) - Task succeeded with attempt attempt_1463523510462_0001_1_00_000000_0
2016-05-17 22:18:30,808 INFO rm.TaskSchedulerEventHandler (TaskSchedulerEventHandler.java:handleEvent(149)) - Processing the event EventType: S_TA_ENDED
2016-05-17 22:18:30,809 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1463523510462_0001_1][Event:TASK_FINISHED]: vertexName=E1896BB2F1BE47659D46CCB6073BE230, taskId=task_1463523510462_0001_1_00_000000, startTime=1463523510743, finishTime=1463523510808, timeTaken=65, status=SUCCEEDED, successfulAttemptID=attempt_1463523510462_0001_1_00_000000_0, diagnostics=, counters=Counters: 28, org.apache.tez.common.counters.DAGCounter, RACK_LOCAL_TASKS=1, File System Counters, FILE_BYTES_READ=110, FILE_BYTES_WRITTEN=727, FILE_READ_OPS=0, FILE_LARGE_READ_OPS=0, FILE_WRITE_OPS=0, org.apache.tez.common.counters.TaskCounter, SPILLED_RECORDS=13, GC_TIME_MILLIS=0, CPU_MILLISECONDS=0, PHYSICAL_MEMORY_BYTES=0, VIRTUAL_MEMORY_BYTES=0, COMMITTED_HEAP_BYTES=513802240, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=13, OUTPUT_BYTES=195, OUTPUT_BYTES_WITH_OVERHEAD=171, OUTPUT_BYTES_PHYSICAL=175, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=0, ADDITIONAL_SPILL_COUNT=0, cascading.flow.SliceCounters, Process_Begin_Time=1463523510790, Process_Duration=3, Process_End_Time=1463523510793, Read_Duration=2, Tuples_Read=13, Tuples_Written=13, Write_Duration=0, cascading.flow.StepCounters, Tuples_Read=13
2016-05-17 22:18:30,810 INFO impl.TaskImpl (TaskImpl.java:handle(862)) - task_1463523510462_0001_1_00_000000 Task Transitioned from RUNNING to SUCCEEDED due to event T_ATTEMPT_SUCCEEDED
2016-05-17 22:18:30,810 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event TASK_FINISHED to history file
2016-05-17 22:18:30,811 INFO impl.VertexImpl (VertexImpl.java:transition(3536)) - Num completed Tasks for vertex_1463523510462_0001_1_00 [E1896BB2F1BE47659D46CCB6073BE230] : 1
2016-05-17 22:18:30,811 INFO impl.VertexImpl (VertexImpl.java:checkVertexForCompletion(1674)) - Vertex succeeded: vertex_1463523510462_0001_1_00 [E1896BB2F1BE47659D46CCB6073BE230]
2016-05-17 22:18:30,816 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:cleanup(712)) - Final Counters : Counters: 28 [[org.apache.tez.common.counters.DAGCounter RACK_LOCAL_TASKS=1][File System Counters FILE_BYTES_READ=110, FILE_BYTES_WRITTEN=727, FILE_READ_OPS=0, FILE_LARGE_READ_OPS=0, FILE_WRITE_OPS=0][org.apache.tez.common.counters.TaskCounter SPILLED_RECORDS=13, GC_TIME_MILLIS=0, CPU_MILLISECONDS=0, PHYSICAL_MEMORY_BYTES=0, VIRTUAL_MEMORY_BYTES=0, COMMITTED_HEAP_BYTES=513802240, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=13, OUTPUT_BYTES=195, OUTPUT_BYTES_WITH_OVERHEAD=171, OUTPUT_BYTES_PHYSICAL=175, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=0, ADDITIONAL_SPILL_COUNT=0][cascading.flow.SliceCounters Process_Begin_Time=1463523510790, Process_Duration=3, Process_End_Time=1463523510793, Read_Duration=2, Tuples_Read=13, Tuples_Written=13, Write_Duration=0][cascading.flow.StepCounters Tuples_Read=13]]
2016-05-17 22:18:30,816 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(164)) - Redirecting log file based on addend:
2016-05-17 22:18:30,816 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(179)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-17 22:18:30,817 INFO task.ContainerReporter (ContainerReporter.java:call(55)) - Attempting to fetch new task for container container_1463523510462_0001_00_000001
2016-05-17 22:18:30,817 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event VERTEX_FINISHED to history file
2016-05-17 22:18:30,818 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1463523510462_0001_1][Event:VERTEX_FINISHED]: vertexName=E1896BB2F1BE47659D46CCB6073BE230, vertexId=vertex_1463523510462_0001_1_00, initRequestedTime=1463523510702, initedTime=1463523510721, startRequestedTime=1463523510723, startedTime=1463523510723, finishTime=1463523510812, timeTaken=89, status=SUCCEEDED, diagnostics=, counters=Counters: 28, org.apache.tez.common.counters.DAGCounter, RACK_LOCAL_TASKS=1, File System Counters, FILE_BYTES_READ=110, FILE_BYTES_WRITTEN=727, FILE_READ_OPS=0, FILE_LARGE_READ_OPS=0, FILE_WRITE_OPS=0, org.apache.tez.common.counters.TaskCounter, SPILLED_RECORDS=13, GC_TIME_MILLIS=0, CPU_MILLISECONDS=0, PHYSICAL_MEMORY_BYTES=0, VIRTUAL_MEMORY_BYTES=0, COMMITTED_HEAP_BYTES=513802240, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=13, OUTPUT_BYTES=195, OUTPUT_BYTES_WITH_OVERHEAD=171, OUTPUT_BYTES_PHYSICAL=175, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=0, ADDITIONAL_SPILL_COUNT=0, cascading.flow.SliceCounters, Process_Begin_Time=1463523510790, Process_Duration=3, Process_End_Time=1463523510793, Read_Duration=2, Tuples_Read=13, Tuples_Written=13, Write_Duration=0, cascading.flow.StepCounters, Tuples_Read=13, vertexStats=firstTaskStartTime=1463523510743, firstTasksToStart=[ task_1463523510462_0001_1_00_000000 ], lastTaskFinishTime=1463523510806, lastTasksToFinish=[ task_1463523510462_0001_1_00_000000 ], minTaskDuration=63, maxTaskDuration=63, avgTaskDuration=63.0, numSuccessfulTasks=1, shortestDurationTasks=[ task_1463523510462_0001_1_00_000000 ], longestDurationTasks=[ task_1463523510462_0001_1_00_000000 ], vertexTaskStats={numFailedTasks=0, numSucceededTasks=1, numKilledTaskAttempts=0, numKilledTasks=0, numFailedTaskAttempts=0, numCompletedTasks=1}
2016-05-17 22:18:30,818 INFO impl.VertexImpl (VertexImpl.java:handle(1564)) - vertex_1463523510462_0001_1_00 [E1896BB2F1BE47659D46CCB6073BE230] transitioned from RUNNING to SUCCEEDED due to event V_TASK_COMPLETED
2016-05-17 22:18:30,818 INFO container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1463523510462_0001_00_000001 transitioned from RUNNING to IDLE via event C_TA_SUCCEEDED
2016-05-17 22:18:30,818 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1463523510462_0001_1][Event:CONTAINER_STOPPED]: containerId=container_1463523510462_0001_00_000001, stoppedTime=1463523510818, exitStatus=0
2016-05-17 22:18:30,818 INFO container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1463523510462_0001_00_000001 transitioned from IDLE to STOP_REQUESTED via event C_STOP_REQUEST
2016-05-17 22:18:30,818 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event CONTAINER_STOPPED to history file
2016-05-17 22:18:30,819 INFO impl.VertexImpl (VertexImpl.java:transition(3477)) - Source task attempt completed for vertex: vertex_1463523510462_0001_1_01 [C2C6CAC5F511435195BF624C09A232D5] attempt: attempt_1463523510462_0001_1_00_000000_0 with state: SUCCEEDED vertexState: RUNNING
2016-05-17 22:18:30,820 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:schedulePendingTasks(608)) - All source tasks assigned. Ramping up 1 remaining tasks for vertex: C2C6CAC5F511435195BF624C09A232D5
2016-05-17 22:18:30,820 INFO impl.VertexImpl (VertexImpl.java:scheduleTasks(1250)) - Routing pending task events for vertex: vertex_1463523510462_0001_1_01 [C2C6CAC5F511435195BF624C09A232D5]
2016-05-17 22:18:30,820 INFO impl.DAGImpl (DAGImpl.java:transition(1747)) - Vertex vertex_1463523510462_0001_1_00 [E1896BB2F1BE47659D46CCB6073BE230] completed., numCompletedVertices=1, numSuccessfulVertices=1, numFailedVertices=0, numKilledVertices=0, numVertices=2
2016-05-17 22:18:30,820 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:stop(244)) - Ignoring stop request for containerId container_1463523510462_0001_00_000001. Relying on regular task shutdown for it to end
2016-05-17 22:18:30,821 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1463523510462_0001_1][Event:TASK_STARTED]: vertexName=C2C6CAC5F511435195BF624C09A232D5, taskId=task_1463523510462_0001_1_01_000000, scheduledTime=1463523510821, launchTime=1463523510821
2016-05-17 22:18:30,821 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event TASK_STARTED to history file
2016-05-17 22:18:30,821 INFO impl.TaskImpl (TaskImpl.java:handle(862)) - task_1463523510462_0001_1_01_000000 Task Transitioned from NEW to SCHEDULED due to event T_SCHEDULE
2016-05-17 22:18:30,822 INFO container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1463523510462_0001_00_000001 transitioned from STOP_REQUESTED to STOPPING via event C_NM_STOP_SENT
2016-05-17 22:18:30,822 INFO impl.DAGSchedulerNaturalOrderControlled (DAGSchedulerNaturalOrderControlled.java:scheduleTask(111)) - Scheduled vertex: vertex_1463523510462_0001_1_01 [C2C6CAC5F511435195BF624C09A232D5]
2016-05-17 22:18:30,822 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:transition(1014)) - remoteTaskSpec:DAGName : [7D2FB5646B554769B5E79B37A91DAD15/FE1AD6A61E6A4290B0EA5055EE3472AA] test/(1/1) ...m/testSimpleBuffer/simple, VertexName: C2C6CAC5F511435195BF624C09A232D5, VertexParallelism: 1, TaskAttemptID:attempt_1463523510462_0001_1_01_000000_0, processorName=cascading.flow.tez.FlowProcessor, inputSpecListSize=1, outputSpecListSize=1, inputSpecList=[{{ sourceVertexName=E1896BB2F1BE47659D46CCB6073BE230, physicalEdgeCount=1, inputClassName=org.apache.tez.runtime.library.input.OrderedGroupedKVInput }}, ], outputSpecList=[{{ destinationVertexName=BA0A5A17BB2E45CC83131A41C1DEC7C7, physicalEdgeCount=0, outputClassName=org.apache.tez.mapreduce.output.MROutput }}, ]
2016-05-17 22:18:30,823 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:handle(692)) - attempt_1463523510462_0001_1_01_000000_0 TaskAttempt Transitioned from NEW to START_WAIT due to event TA_SCHEDULE
2016-05-17 22:18:30,823 INFO rm.TaskSchedulerEventHandler (TaskSchedulerEventHandler.java:handleEvent(149)) - Processing the event EventType: S_TA_LAUNCH_REQUEST
2016-05-17 22:18:30,824 INFO container.AMContainerImpl (AMContainerImpl.java:transition(521)) - Sending Launch Request for Container with id: container_1463523510462_0001_00_000002
2016-05-17 22:18:30,824 INFO container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1463523510462_0001_00_000002 transitioned from ALLOCATED to LAUNCHING via event C_LAUNCH_REQUEST
2016-05-17 22:18:30,826 INFO task.TezChild (TezChild.java:newTezChild(432)) - PID, containerIdentifier: null, container_1463523510462_0001_00_000002
2016-05-17 22:18:30,826 WARN impl.MetricsSystemImpl (MetricsSystemImpl.java:init(145)) - DAGAppMaster metrics system already initialized!
2016-05-17 22:18:30,827 INFO container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1463523510462_0001_00_000002 transitioned from LAUNCHING to IDLE via event C_LAUNCHED
2016-05-17 22:18:30,827 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event CONTAINER_LAUNCHED to history file
2016-05-17 22:18:30,827 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1463523510462_0001_1][Event:CONTAINER_LAUNCHED]: containerId=container_1463523510462_0001_00_000002, launchTime=1463523510827
2016-05-17 22:18:30,828 INFO app.TaskAttemptListenerImpTezDag (TaskAttemptListenerImpTezDag.java:getTask(213)) - Container with id: container_1463523510462_0001_00_000001 is valid, but no longer registered, and will be killed
2016-05-17 22:18:30,828 INFO task.ContainerReporter (ContainerReporter.java:call(65)) - Got TaskUpdate for containerId= container_1463523510462_0001_00_000001: 11 ms after starting to poll. TaskInfo: shouldDie: true
2016-05-17 22:18:30,828 INFO task.TezChild (TezChild.java:run(207)) - ContainerTask returned shouldDie=true for container container_1463523510462_0001_00_000001, Exiting
2016-05-17 22:18:30,828 INFO task.TezChild (TezChild.java:shutdown(343)) - Shutdown invoked for container container_1463523510462_0001_00_000001
2016-05-17 22:18:30,828 INFO task.TezChild (TezChild.java:shutdown(345)) - Shutting down container container_1463523510462_0001_00_000001
2016-05-17 22:18:30,828 INFO task.ContainerReporter (ContainerReporter.java:call(55)) - Attempting to fetch new task for container container_1463523510462_0001_00_000002
2016-05-17 22:18:30,830 INFO container.AMContainerImpl (AMContainerImpl.java:transition(827)) - Assigned taskAttempt + [attempt_1463523510462_0001_1_01_000000_0] to container: [container_1463523510462_0001_00_000002]
2016-05-17 22:18:30,831 INFO container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1463523510462_0001_00_000002 transitioned from IDLE to RUNNING via event C_PULL_TA
2016-05-17 22:18:30,831 INFO app.TaskAttemptListenerImpTezDag (TaskAttemptListenerImpTezDag.java:getTask(240)) - Container with id: container_1463523510462_0001_00_000002 given task: attempt_1463523510462_0001_1_01_000000_0
2016-05-17 22:18:30,831 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(270)) - ContainerExecutionResult for: container_1463523510462_0001_00_000001 = ContainerExecutionResult{exitStatus=SUCCESS, throwable=null, errorMessage='Asked to die by the AM'}
2016-05-17 22:18:30,831 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(274)) - Container: container_1463523510462_0001_00_000001 completed successfully
2016-05-17 22:18:30,831 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:transition(1146)) - TaskAttempt: [attempt_1463523510462_0001_1_01_000000_0] started. Is using containerId: [container_1463523510462_0001_00_000002] on NM: [127.0.0.1:0]
2016-05-17 22:18:30,831 INFO task.ContainerReporter (ContainerReporter.java:call(65)) - Got TaskUpdate for containerId= container_1463523510462_0001_00_000002: 0 ms after starting to poll. TaskInfo: shouldDie: false, currentTaskAttemptId: attempt_1463523510462_0001_1_01_000000_0
2016-05-17 22:18:30,831 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event TASK_ATTEMPT_STARTED to history file
2016-05-17 22:18:30,832 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1463523510462_0001_1][Event:TASK_ATTEMPT_STARTED]: vertexName=C2C6CAC5F511435195BF624C09A232D5, taskAttemptId=attempt_1463523510462_0001_1_01_000000_0, startTime=1463523510831, containerId=container_1463523510462_0001_00_000002, nodeId=127.0.0.1:0, inProgressLogs=127.0.0.1:0/node/containerlogs/container_1463523510462_0001_00_000002/teamcity, completedLogs=
2016-05-17 22:18:30 Starting to run new task attempt: attempt_1463523510462_0001_1_01_000000_0
2016-05-17 22:18:30,832 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(164)) - Redirecting log file based on addend: attempt_1463523510462_0001_1_01_000000_0
2016-05-17 22:18:30,832 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:handle(692)) - attempt_1463523510462_0001_1_01_000000_0 TaskAttempt Transitioned from START_WAIT to RUNNING due to event TA_STARTED_REMOTELY
2016-05-17 22:18:30,832 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(179)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-17 22:18:30,832 INFO container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1463523510462_0001_00_000001 transitioned from STOPPING to COMPLETED via event C_COMPLETED
2016-05-17 22:18:30,832 INFO task.TezChild (TezChild.java:handleNewTaskCredentials(273)) - Refreshing UGI since Credentials have changed
2016-05-17 22:18:30,833 INFO task.TezChild (TezChild.java:handleNewTaskCredentials(276)) - Credentials : #Tokens=1, #SecretKeys=0
2016-05-17 22:18:30,833 INFO task.TezChild (TezChild.java:handleNewTaskLocalResources(302)) - Localizing additional local resources for Task : {}
2016-05-17 22:18:30,833 INFO task.TezChild (TezChild.java:handleNewTaskLocalResources(312)) - Done localizing additional resources
2016-05-17 22:18:30,833 INFO metrics.TaskCounterUpdater (TaskCounterUpdater.java:initResourceCalculatorPlugin(148)) - Using ResourceCalculatorProcessTree : [ ]
2016-05-17 22:18:30,833 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:<init>(149)) - Initializing LogicalIOProcessorRuntimeTask with TaskSpec: DAGName : [7D2FB5646B554769B5E79B37A91DAD15/FE1AD6A61E6A4290B0EA5055EE3472AA] test/(1/1) ...m/testSimpleBuffer/simple, VertexName: C2C6CAC5F511435195BF624C09A232D5, VertexParallelism: 1, TaskAttemptID:attempt_1463523510462_0001_1_01_000000_0, processorName=cascading.flow.tez.FlowProcessor, inputSpecListSize=1, outputSpecListSize=1, inputSpecList=[{{ sourceVertexName=E1896BB2F1BE47659D46CCB6073BE230, physicalEdgeCount=1, inputClassName=org.apache.tez.runtime.library.input.OrderedGroupedKVInput }}, ], outputSpecList=[{{ destinationVertexName=BA0A5A17BB2E45CC83131A41C1DEC7C7, physicalEdgeCount=0, outputClassName=org.apache.tez.mapreduce.output.MROutput }}, ]
2016-05-17 22:18:30,833 INFO resources.MemoryDistributor (MemoryDistributor.java:<init>(85)) - InitialMemoryDistributor (isEnabled=true) invoked with: numInputs=1, numOutputs=1, JVM.maxFree=704643072
2016-05-17 22:18:30,834 INFO impl.TaskImpl (TaskImpl.java:handle(862)) - task_1463523510462_0001_1_01_000000 Task Transitioned from SCHEDULED to RUNNING due to event T_ATTEMPT_LAUNCHED
2016-05-17 22:18:30,836 INFO task.TezTaskRunner (TezTaskRunner.java:run(172)) - Initializing task, taskAttemptId=attempt_1463523510462_0001_1_01_000000_0
2016-05-17 22:18:30,836 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(191)) - Initializing LogicalProcessorIORuntimeTask
2016-05-17 22:18:30,836 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(195)) - Creating processor, processorClassName=cascading.flow.tez.FlowProcessor
2016-05-17 22:18:30,838 INFO impl.TaskImpl (TaskImpl.java:getTaskAttemptTezEvents(481)) - TaskAttempt:attempt_1463523510462_0001_1_01_000000_0 sent events: (0-1)
2016-05-17 22:18:30,840 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(389)) - Initializing Input using InputSpec: { sourceVertexName=E1896BB2F1BE47659D46CCB6073BE230, physicalEdgeCount=1, inputClassName=org.apache.tez.runtime.library.input.OrderedGroupedKVInput }
2016-05-17 22:18:30,841 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(438)) - Initializing Output using OutputSpec: { destinationVertexName=BA0A5A17BB2E45CC83131A41C1DEC7C7, physicalEdgeCount=0, outputClassName=org.apache.tez.mapreduce.output.MROutput }
2016-05-17 22:18:30,840 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initializeLogicalIOProcessor(488)) - Initializing processor, processorClassName=cascading.flow.tez.FlowProcessor
2016-05-17 22:18:30,841 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:createInput(537)) - Creating Input
2016-05-17 22:18:30,841 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(397)) - Initializing Input with src edge: E1896BB2F1BE47659D46CCB6073BE230
2016-05-17 22:18:30,843 INFO tez.FlowProcessor (FlowProcessor.java:initialize(88)) - cascading version: 3.1.0
2016-05-17 22:18:30,843 INFO orderedgrouped.MergeManager (MergeManager.java:getInitialMemoryRequirement(310)) - Initial Shuffle Memory Required: 634178752, based on INPUT_BUFFER_factor: 0.9
2016-05-17 22:18:30,844 INFO orderedgrouped.MergeManager (MergeManager.java:getInitialMemoryRequirement(320)) - Initial Memory required for final merged output: 0, using factor: 0.0
2016-05-17 22:18:30,844 INFO resources.MemoryDistributor$RequestorInfo (MemoryDistributor.java:<init>(233)) - Received request: 634178752, type: INPUT, componentVertexName: E1896BB2F1BE47659D46CCB6073BE230
2016-05-17 22:18:30,844 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(403)) - Initialized Input with src edge: E1896BB2F1BE47659D46CCB6073BE230
2016-05-17 22:18:30,848 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:createOutput(565)) - Creating Output
2016-05-17 22:18:30,849 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(446)) - Initializing Output with dest edge: BA0A5A17BB2E45CC83131A41C1DEC7C7
2016-05-17 22:18:30,849 INFO output.MROutput (MROutput.java:initialize(347)) - Initializing Simple Output
2016-05-17 22:18:30,849 INFO resources.MemoryDistributor$RequestorInfo (MemoryDistributor.java:<init>(233)) - Received request: 0, type: OUTPUT, componentVertexName: BA0A5A17BB2E45CC83131A41C1DEC7C7
2016-05-17 22:18:30,853 INFO tez.FlowProcessor (FlowProcessor.java:initialize(94)) - flow node id: C2C6CAC5F511435195BF624C09A232D5, ordinal: 1
2016-05-17 22:18:30,853 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: C2C6CAC5F511435195BF624C09A232D5, mem on start (mb), free: 142, total: 490, max: 672
2016-05-17 22:18:30,853 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initializeLogicalIOProcessor(491)) - Initialized processor, processorClassName=cascading.flow.tez.FlowProcessor
2016-05-17 22:18:30,853 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(220)) - Waiting for 2 initializers to finish
2016-05-17 22:18:30,853 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(220)) - Waiting for 1 initializers to finish
2016-05-17 22:18:30,861 INFO output.MROutput (MROutput.java:initialize(418)) - Initialized Simple Output, using_new_api: false
2016-05-17 22:18:30,861 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(452)) - Initialized Output with dest edge: BA0A5A17BB2E45CC83131A41C1DEC7C7
2016-05-17 22:18:30,861 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(233)) - All initializers finished
2016-05-17 22:18:30,862 INFO resources.MemoryDistributor (MemoryDistributor.java:makeInitialAllocations(124)) - Using Allocator class: org.apache.tez.runtime.library.resources.WeightedScalingMemoryDistributor
2016-05-17 22:18:30,862 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:initialProcessMemoryRequestContext(171)) - ScaleFactor: 12, for type: SORTED_MERGED_INPUT
2016-05-17 22:18:30,862 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:getRequestTypeForClass(197)) - Falling back to RequestType.OTHER for class: org.apache.tez.mapreduce.output.MROutput
2016-05-17 22:18:30,862 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:initialProcessMemoryRequestContext(171)) - ScaleFactor: 1, for type: OTHER
2016-05-17 22:18:30,862 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:computeReservedFraction(262)) - InitialReservationFraction=0.3, AdditionalReservationFractionForIOs=0.03, finalReserveFractionUsed=0.32999999999999996
2016-05-17 22:18:30,862 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:assignMemory(126)) - Scaling Requests. NumRequests: 2, numScaledRequests: 13, TotalRequested: 634178752, TotalRequestedScaled: 5.853957710769231E8, TotalJVMHeap: 704643072, TotalAvailable: 472110858, TotalRequested/TotalJVMHeap:0.90
2016-05-17 22:18:30,863 INFO resources.MemoryDistributor (MemoryDistributor.java:makeInitialAllocations(138)) - Informing: INPUT, E1896BB2F1BE47659D46CCB6073BE230, org.apache.tez.runtime.library.input.OrderedGroupedKVInput: requested=634178752, allocated=472110858
2016-05-17 22:18:30,863 INFO resources.MemoryDistributor (MemoryDistributor.java:makeInitialAllocations(138)) - Informing: OUTPUT, BA0A5A17BB2E45CC83131A41C1DEC7C7, org.apache.tez.mapreduce.output.MROutput: requested=0, allocated=0
2016-05-17 22:18:30,863 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(255)) - Starting Inputs/Outputs
2016-05-17 22:18:30,863 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(268)) - Input: E1896BB2F1BE47659D46CCB6073BE230 being auto started by the framework. Subsequent instances will not be auto-started
2016-05-17 22:18:30,863 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(289)) - Num IOs determined for AutoStart: 1
2016-05-17 22:18:30,863 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(291)) - Waiting for 1 IOs to start
2016-05-17 22:18:30,863 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(419)) - Starting Input with src edge: E1896BB2F1BE47659D46CCB6073BE230
2016-05-17 22:18:30,864 INFO common.TezRuntimeUtils (TezRuntimeUtils.java:instantiateCombiner(65)) - No combiner specified via tez.runtime.combiner.class. Combiner will not be used
2016-05-17 22:18:30,865 INFO orderedgrouped.Shuffle (Shuffle.java:<init>(182)) - Shuffle assigned with 1 inputs, codec: NoneifileReadAhead: true
2016-05-17 22:18:30,865 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:<init>(158)) - ShuffleScheduler running for sourceVertex: E1896BB2F1BE47659D46CCB6073BE230 with configuration: maxFetchFailuresBeforeReporting=5, reportReadErrorImmediately=true, maxFailedUniqueFetches=1, abortFailureLimit=30, maxMapRuntime=0, maxTaskOutputAtOnce=20
2016-05-17 22:18:30,866 INFO orderedgrouped.MergeManager (MergeManager.java:<init>(221)) - InitialRequest: ShuffleMem=634178752, postMergeMem=0, RuntimeTotalAvailable=472110858. Updated to: ShuffleMem=472110858, postMergeMem: 0
2016-05-17 22:18:30,866 INFO orderedgrouped.MergeManager (MergeManager.java:<init>(252)) - MergerManager: memoryLimit=472110858, maxSingleShuffleLimit=118027712, mergeThreshold=424899744, ioSortFactor=100, memToMemMergeOutputsThreshold=100
2016-05-17 22:18:30,866 INFO orderedgrouped.Shuffle (Shuffle.java:<init>(233)) - Num fetchers being started: 1
2016-05-17 22:18:30,869 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(421)) - Started Input with src edge: E1896BB2F1BE47659D46CCB6073BE230
2016-05-17 22:18:30,869 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(304)) - AutoStartComplete
2016-05-17 22:18:30,869 INFO task.TezTaskRunner (TezTaskRunner.java:run(175)) - Running task, taskAttemptId=attempt_1463523510462_0001_1_01_000000_0
2016-05-17 22:18:30,873 INFO orderedgrouped.ShuffleInputEventHandlerOrderedGrouped (ShuffleInputEventHandlerOrderedGrouped.java:processDataMovementEvent(84)) - DME srcIdx: 0, targetIdx: 0, attemptNum: 0, payload: [host: ip-10-123-148-208, port: 0, pathComponent: attempt_1463523510462_0001_1_00_000000_0_10015, runDuration: 47984, ]
2016-05-17 22:18:30,873 INFO graph.Hadoop2TezStreamGraph (Hadoop2TezStreamGraph.java:buildGraph(151)) - using streamed source: GroupBy(test)[by:[{1}:'num']]
2016-05-17 22:18:30,874 INFO tez.FlowProcessor (FlowProcessor.java:run(121)) - sourcing from: GroupBy(test)[by:[{1}:'num']] streamed: true, id: E43200532F7043DA95777CF1D2395C76
2016-05-17 22:18:30,874 INFO tez.FlowProcessor (FlowProcessor.java:run(124)) - sinking to: Hfs["TextLine[['line']->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.5.x-tez-0.6.x/build/test/output/hadoop2-tez/bufferpipesplatform/testSimpleBuffer/simple"], id: BA0A5A17BB2E45CC83131A41C1DEC7C7
2016-05-17 22:18:30,875 INFO element.TezSinkStage (TezSinkStage.java:prepare(57)) - calling MROutput#start() on: Hfs["TextLine[['line']->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.5.x-tez-0.6.x/build/test/output/hadoop2-tez/bufferpipesplatform/testSimpleBuffer/simple"]
2016-05-17 22:18:30,877 INFO element.TezGroupGate (TezGroupGate.java:prepare(90)) - calling OrderedGroupedKVInput#start() on: GroupBy(test)[by:[{1}:'num']] E43200532F7043DA95777CF1D2395C76, for 1 inputs
2016-05-17 22:18:30,878 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:logIndividualFetchComplete(250)) - Completed fetch for attempt: InputAttemptIdentifier [inputIdentifier=InputIdentifier [inputIndex=0], attemptNumber=0, pathComponent=attempt_1463523510462_0001_1_00_000000_0_10015] to DISK_DIRECT, CompressedSize=175, DecompressedSize=171,EndTime=1463523510877, TimeTaken=2, Rate=0.08 MB/s
2016-05-17 22:18:30,878 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:copySucceeded(213)) - All inputs fetched for input vertex : E1896BB2F1BE47659D46CCB6073BE230
2016-05-17 22:18:30,878 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:logProgress(263)) - copy(1 of 1. Transfer rate (CumulativeDataFetched/TimeSinceInputStarted)) 0.00 MB/s)
2016-05-17 22:18:30,878 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:freeHost(529)) - ip-10-123-148-208:0 freed by fetcher [E1896BB2F1BE47659D46CCB6073BE230] #3 in 3ms
2016-05-17 22:18:30,879 INFO orderedgrouped.Shuffle (Shuffle.java:cleanupFetchers(394)) - Shutdown..fetcher [E1896BB2F1BE47659D46CCB6073BE230] #3, status:false, isInterrupted:false
2016-05-17 22:18:30,879 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(854)) - finalMerge called with 0 in-memory map-outputs and 1 on-disk map-outputs
2016-05-17 22:18:30,880 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(951)) - Merging 1 files, 175 bytes from disk
2016-05-17 22:18:30,880 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(966)) - Merging 0 segments, 0 bytes from memory into reduce
2016-05-17 22:18:30,880 INFO impl.TezMerger (TezMerger.java:merge(560)) - Merging 1 sorted segments
2016-05-17 22:18:30,880 INFO impl.TezMerger (TezMerger.java:merge(665)) - Down to the last merge-pass, with 1 segments left of total size: 158 bytes
2016-05-17 22:18:30,880 INFO orderedgrouped.Shuffle (Shuffle.java:call(381)) - merge complete for input vertex : E1896BB2F1BE47659D46CCB6073BE230
2016-05-17 22:18:30,880 INFO tez.FlowProcessor (FlowProcessor.java:waitForInputsReady(202)) - flow node id: C2C6CAC5F511435195BF624C09A232D5, all 1 inputs ready in: 00:00:00.003
2016-05-17 22:18:30,881 INFO orderedgrouped.Shuffle (Shuffle.java:onSuccess(488)) - Shuffle Runner thread complete
2016-05-17 22:18:30,882 INFO common.ValuesIterator (ValuesIterator.java:<init>(83)) - keyDeserializer=cascading.tuple.hadoop.io.KeyTupleDeserializer@b44f693; valueDeserializer=cascading.tuple.hadoop.io.ValueTupleDeserializer@5c9832ac
2016-05-17 22:18:30,885 INFO output.MROutput (MROutput.java:flush(534)) - Flushing Simple Output
2016-05-17 22:18:30,885 INFO output.MROutput (MROutput.java:flush(544)) - Flushed Simple Output
2016-05-17 22:18:30,886 INFO app.TaskAttemptListenerImpTezDag (TaskAttemptListenerImpTezDag.java:canCommit(353)) - Commit go/no-go request from attempt_1463523510462_0001_1_01_000000_0
2016-05-17 22:18:30,886 INFO impl.TaskImpl (TaskImpl.java:canCommit(750)) - attempt_1463523510462_0001_1_01_000000_0 given a go for committing the task output.
2016-05-17 22:18:30,887 INFO task.TezTaskRunner (TezTaskRunner.java:run(177)) - Closing task, taskAttemptId=attempt_1463523510462_0001_1_01_000000_0
2016-05-17 22:18:30,887 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: C2C6CAC5F511435195BF624C09A232D5, mem on close (mb), free: 138, total: 490, max: 672
2016-05-17 22:18:30,888 INFO orderedgrouped.Shuffle (Shuffle.java:shutdown(320)) - Shutting down Shuffle for source: E1896BB2F1BE47659D46CCB6073BE230
2016-05-17 22:18:30,890 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(854)) - finalMerge called with 0 in-memory map-outputs and 0 on-disk map-outputs
2016-05-17 22:18:30,890 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(951)) - Merging 0 files, 0 bytes from disk
2016-05-17 22:18:30,890 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(966)) - Merging 0 segments, 0 bytes from memory into reduce
2016-05-17 22:18:30,891 INFO impl.TezMerger (TezMerger.java:merge(560)) - Merging 0 sorted segments
2016-05-17 22:18:30,891 INFO impl.TezMerger (TezMerger.java:merge(562)) - Nothing to merge. Returning an empty iterator
2016-05-17 22:18:30 Completed running task attempt: attempt_1463523510462_0001_1_01_000000_0
2016-05-17 22:18:30,892 INFO task.TezTaskRunner (TezTaskRunner.java:run(181)) - Task completed, taskAttemptId=attempt_1463523510462_0001_1_01_000000_0, fatalErrorOccurred=false
2016-05-17 22:18:30,893 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event TASK_ATTEMPT_FINISHED to history file
2016-05-17 22:18:30,894 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:cleanup(712)) - Final Counters : Counters: 46 [[File System Counters FILE_BYTES_READ=207, FILE_BYTES_WRITTEN=387, FILE_READ_OPS=0, FILE_LARGE_READ_OPS=0, FILE_WRITE_OPS=0][org.apache.tez.common.counters.TaskCounter REDUCE_INPUT_GROUPS=4, REDUCE_INPUT_RECORDS=13, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=13, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, GC_TIME_MILLIS=0, CPU_MILLISECONDS=0, PHYSICAL_MEMORY_BYTES=0, VIRTUAL_MEMORY_BYTES=0, COMMITTED_HEAP_BYTES=513802240, OUTPUT_RECORDS=23, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=175, SHUFFLE_BYTES=175, SHUFFLE_BYTES_DECOMPRESSED=171, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=175, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=13, MERGE_PHASE_TIME=15, FIRST_EVENT_RECEIVED=9, LAST_EVENT_RECEIVED=9][Shuffle Errors BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0][cascading.flow.SliceCounters Process_Begin_Time=1463523510881, Process_Duration=6, Process_End_Time=1463523510887, Read_Duration=1, Tuples_Read=13, Tuples_Written=23, Write_Duration=0][cascading.flow.StepCounters Tuples_Written=23]]
2016-05-17 22:18:30,894 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(164)) - Redirecting log file based on addend:
2016-05-17 22:18:30,894 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(179)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-17 22:18:30,895 INFO task.ContainerReporter (ContainerReporter.java:call(55)) - Attempting to fetch new task for container container_1463523510462_0001_00_000002
2016-05-17 22:18:30,895 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1463523510462_0001_1][Event:TASK_ATTEMPT_FINISHED]: vertexName=C2C6CAC5F511435195BF624C09A232D5, taskAttemptId=attempt_1463523510462_0001_1_01_000000_0, startTime=1463523510831, finishTime=1463523510893, timeTaken=62, status=SUCCEEDED, errorEnum=, diagnostics=, counters=Counters: 46, File System Counters, FILE_BYTES_READ=207, FILE_BYTES_WRITTEN=387, FILE_READ_OPS=0, FILE_LARGE_READ_OPS=0, FILE_WRITE_OPS=0, org.apache.tez.common.counters.TaskCounter, REDUCE_INPUT_GROUPS=4, REDUCE_INPUT_RECORDS=13, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=13, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, GC_TIME_MILLIS=0, CPU_MILLISECONDS=0, PHYSICAL_MEMORY_BYTES=0, VIRTUAL_MEMORY_BYTES=0, COMMITTED_HEAP_BYTES=513802240, OUTPUT_RECORDS=23, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=175, SHUFFLE_BYTES=175, SHUFFLE_BYTES_DECOMPRESSED=171, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=175, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=13, MERGE_PHASE_TIME=15, FIRST_EVENT_RECEIVED=9, LAST_EVENT_RECEIVED=9, Shuffle Errors, BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0, cascading.flow.SliceCounters, Process_Begin_Time=1463523510881, Process_Duration=6, Process_End_Time=1463523510887, Read_Duration=1, Tuples_Read=13, Tuples_Written=23, Write_Duration=0, cascading.flow.StepCounters, Tuples_Written=23
2016-05-17 22:18:30,896 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:handle(692)) - attempt_1463523510462_0001_1_01_000000_0 TaskAttempt Transitioned from RUNNING to SUCCEEDED due to event TA_DONE
2016-05-17 22:18:30,896 INFO impl.TaskImpl (TaskImpl.java:transition(1052)) - Task succeeded with attempt attempt_1463523510462_0001_1_01_000000_0
2016-05-17 22:18:30,897 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event TASK_FINISHED to history file
2016-05-17 22:18:30,897 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1463523510462_0001_1][Event:TASK_FINISHED]: vertexName=C2C6CAC5F511435195BF624C09A232D5, taskId=task_1463523510462_0001_1_01_000000, startTime=1463523510831, finishTime=1463523510896, timeTaken=65, status=SUCCEEDED, successfulAttemptID=attempt_1463523510462_0001_1_01_000000_0, diagnostics=, counters=Counters: 46, File System Counters, FILE_BYTES_READ=207, FILE_BYTES_WRITTEN=387, FILE_READ_OPS=0, FILE_LARGE_READ_OPS=0, FILE_WRITE_OPS=0, org.apache.tez.common.counters.TaskCounter, REDUCE_INPUT_GROUPS=4, REDUCE_INPUT_RECORDS=13, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=13, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, GC_TIME_MILLIS=0, CPU_MILLISECONDS=0, PHYSICAL_MEMORY_BYTES=0, VIRTUAL_MEMORY_BYTES=0, COMMITTED_HEAP_BYTES=513802240, OUTPUT_RECORDS=23, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=175, SHUFFLE_BYTES=175, SHUFFLE_BYTES_DECOMPRESSED=171, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=175, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=13, MERGE_PHASE_TIME=15, FIRST_EVENT_RECEIVED=9, LAST_EVENT_RECEIVED=9, Shuffle Errors, BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0, cascading.flow.SliceCounters, Process_Begin_Time=1463523510881, Process_Duration=6, Process_End_Time=1463523510887, Read_Duration=1, Tuples_Read=13, Tuples_Written=23, Write_Duration=0, cascading.flow.StepCounters, Tuples_Written=23
2016-05-17 22:18:30,897 INFO impl.TaskImpl (TaskImpl.java:handle(862)) - task_1463523510462_0001_1_01_000000 Task Transitioned from RUNNING to SUCCEEDED due to event T_ATTEMPT_SUCCEEDED
2016-05-17 22:18:30,898 INFO rm.TaskSchedulerEventHandler (TaskSchedulerEventHandler.java:handleEvent(149)) - Processing the event EventType: S_TA_ENDED
2016-05-17 22:18:30,898 INFO impl.VertexImpl (VertexImpl.java:transition(3536)) - Num completed Tasks for vertex_1463523510462_0001_1_01 [C2C6CAC5F511435195BF624C09A232D5] : 1
2016-05-17 22:18:30,899 INFO impl.VertexImpl (VertexImpl.java:checkVertexForCompletion(1674)) - Vertex succeeded: vertex_1463523510462_0001_1_01 [C2C6CAC5F511435195BF624C09A232D5]
2016-05-17 22:18:30,900 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event VERTEX_FINISHED to history file
2016-05-17 22:18:30,900 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1463523510462_0001_1][Event:VERTEX_FINISHED]: vertexName=C2C6CAC5F511435195BF624C09A232D5, vertexId=vertex_1463523510462_0001_1_01, initRequestedTime=1463523510705, initedTime=1463523510719, startRequestedTime=1463523510728, startedTime=1463523510728, finishTime=1463523510899, timeTaken=171, status=SUCCEEDED, diagnostics=, counters=Counters: 46, File System Counters, FILE_BYTES_READ=207, FILE_BYTES_WRITTEN=387, FILE_READ_OPS=0, FILE_LARGE_READ_OPS=0, FILE_WRITE_OPS=0, org.apache.tez.common.counters.TaskCounter, REDUCE_INPUT_GROUPS=4, REDUCE_INPUT_RECORDS=13, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=13, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, GC_TIME_MILLIS=0, CPU_MILLISECONDS=0, PHYSICAL_MEMORY_BYTES=0, VIRTUAL_MEMORY_BYTES=0, COMMITTED_HEAP_BYTES=513802240, OUTPUT_RECORDS=23, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=175, SHUFFLE_BYTES=175, SHUFFLE_BYTES_DECOMPRESSED=171, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=175, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=13, MERGE_PHASE_TIME=15, FIRST_EVENT_RECEIVED=9, LAST_EVENT_RECEIVED=9, Shuffle Errors, BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0, cascading.flow.SliceCounters, Process_Begin_Time=1463523510881, Process_Duration=6, Process_End_Time=1463523510887, Read_Duration=1, Tuples_Read=13, Tuples_Written=23, Write_Duration=0, cascading.flow.StepCounters, Tuples_Written=23, vertexStats=firstTaskStartTime=1463523510831, firstTasksToStart=[ task_1463523510462_0001_1_01_000000 ], lastTaskFinishTime=1463523510893, lastTasksToFinish=[ task_1463523510462_0001_1_01_000000 ], minTaskDuration=62, maxTaskDuration=62, avgTaskDuration=62.0, numSuccessfulTasks=1, shortestDurationTasks=[ task_1463523510462_0001_1_01_000000 ], longestDurationTasks=[ task_1463523510462_0001_1_01_000000 ], vertexTaskStats={numFailedTasks=0, numSucceededTasks=1, numKilledTaskAttempts=0, numKilledTasks=0, numFailedTaskAttempts=0, numCompletedTasks=1}
2016-05-17 22:18:30,901 INFO impl.VertexImpl (VertexImpl.java:handle(1564)) - vertex_1463523510462_0001_1_01 [C2C6CAC5F511435195BF624C09A232D5] transitioned from RUNNING to SUCCEEDED due to event V_TASK_COMPLETED
2016-05-17 22:18:30,901 INFO container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1463523510462_0001_00_000002 transitioned from RUNNING to IDLE via event C_TA_SUCCEEDED
2016-05-17 22:18:30,902 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1463523510462_0001_1][Event:CONTAINER_STOPPED]: containerId=container_1463523510462_0001_00_000002, stoppedTime=1463523510901, exitStatus=0
2016-05-17 22:18:30,902 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event CONTAINER_STOPPED to history file
2016-05-17 22:18:30,902 INFO container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1463523510462_0001_00_000002 transitioned from IDLE to STOP_REQUESTED via event C_STOP_REQUEST
2016-05-17 22:18:30,902 INFO impl.DAGImpl (DAGImpl.java:transition(1747)) - Vertex vertex_1463523510462_0001_1_01 [C2C6CAC5F511435195BF624C09A232D5] completed., numCompletedVertices=2, numSuccessfulVertices=2, numFailedVertices=0, numKilledVertices=0, numVertices=2
2016-05-17 22:18:30,902 INFO impl.DAGImpl (DAGImpl.java:commitOrAbortOutputs(831)) - Calling DAG commit/abort for dag: dag_1463523510462_0001_1
2016-05-17 22:18:30,903 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1463523510462_0001_1][Event:DAG_COMMIT_STARTED]: dagID=dag_1463523510462_0001_1
2016-05-17 22:18:30,903 INFO impl.DAGImpl (DAGImpl.java:commitOrAbortOutputs(892)) - Committing output: BA0A5A17BB2E45CC83131A41C1DEC7C7 for vertex: vertex_1463523510462_0001_1_01 [C2C6CAC5F511435195BF624C09A232D5]
2016-05-17 22:18:30,906 INFO app.TaskAttemptListenerImpTezDag (TaskAttemptListenerImpTezDag.java:getTask(213)) - Container with id: container_1463523510462_0001_00_000002 is valid, but no longer registered, and will be killed
2016-05-17 22:18:30,906 INFO task.ContainerReporter (ContainerReporter.java:call(65)) - Got TaskUpdate for containerId= container_1463523510462_0001_00_000002: 10 ms after starting to poll. TaskInfo: shouldDie: true
2016-05-17 22:18:30,906 INFO task.TezChild (TezChild.java:run(207)) - ContainerTask returned shouldDie=true for container container_1463523510462_0001_00_000002, Exiting
2016-05-17 22:18:30,906 INFO task.TezChild (TezChild.java:shutdown(343)) - Shutdown invoked for container container_1463523510462_0001_00_000002
2016-05-17 22:18:30,906 INFO task.TezChild (TezChild.java:shutdown(345)) - Shutting down container container_1463523510462_0001_00_000002
2016-05-17 22:18:30,907 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(270)) - ContainerExecutionResult for: container_1463523510462_0001_00_000002 = ContainerExecutionResult{exitStatus=SUCCESS, throwable=null, errorMessage='Asked to die by the AM'}
2016-05-17 22:18:30,907 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(274)) - Container: container_1463523510462_0001_00_000002 completed successfully
2016-05-17 22:18:30,913 INFO impl.DAGImpl (DAGImpl.java:commitOrAbortOutputs(871)) - No output committers for vertex: vertex_1463523510462_0001_1_00 [E1896BB2F1BE47659D46CCB6073BE230]
2016-05-17 22:18:30,916 INFO recovery.RecoveryService (RecoveryService.java:handle(306)) - DAG completed, dagId=dag_1463523510462_0001_1, queueSize=0
2016-05-17 22:18:30,916 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event DAG_FINISHED to history file
2016-05-17 22:18:30,916 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1463523510462_0001_1][Event:DAG_FINISHED]: dagId=dag_1463523510462_0001_1, startTime=1463523510700, finishTime=1463523510913, timeTaken=213, status=SUCCEEDED, diagnostics=, counters=Counters: 55, org.apache.tez.common.counters.DAGCounter, NUM_SUCCEEDED_TASKS=2, TOTAL_LAUNCHED_TASKS=2, RACK_LOCAL_TASKS=1, File System Counters, FILE_BYTES_READ=317, FILE_BYTES_WRITTEN=1114, FILE_READ_OPS=0, FILE_LARGE_READ_OPS=0, FILE_WRITE_OPS=0, org.apache.tez.common.counters.TaskCounter, REDUCE_INPUT_GROUPS=4, REDUCE_INPUT_RECORDS=13, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=26, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, GC_TIME_MILLIS=0, CPU_MILLISECONDS=0, PHYSICAL_MEMORY_BYTES=0, VIRTUAL_MEMORY_BYTES=0, COMMITTED_HEAP_BYTES=1027604480, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=36, OUTPUT_BYTES=195, OUTPUT_BYTES_WITH_OVERHEAD=171, OUTPUT_BYTES_PHYSICAL=175, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=175, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_BYTES=175, SHUFFLE_BYTES_DECOMPRESSED=171, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=175, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=13, MERGE_PHASE_TIME=15, FIRST_EVENT_RECEIVED=9, LAST_EVENT_RECEIVED=9, Shuffle Errors, BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0, cascading.flow.SliceCounters, Process_Begin_Time=2927047021671, Process_Duration=9, Process_End_Time=2927047021680, Read_Duration=3, Tuples_Read=26, Tuples_Written=36, Write_Duration=0, cascading.flow.StepCounters, Tuples_Read=13, Tuples_Written=23
2016-05-17 22:18:30,916 INFO impl.DAGImpl (DAGImpl.java:finished(1180)) - DAG: dag_1463523510462_0001_1 finished with state: SUCCEEDED
2016-05-17 22:18:30,916 INFO impl.DAGImpl (DAGImpl.java:handle(973)) - dag_1463523510462_0001_1 transitioned from RUNNING to SUCCEEDED due to event DAG_VERTEX_COMPLETED
2016-05-17 22:18:30,917 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:stop(242)) - Ignoring stop request for containerId: container_1463523510462_0001_00_000002
2016-05-17 22:18:30,917 INFO container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1463523510462_0001_00_000002 transitioned from STOP_REQUESTED to COMPLETED via event C_COMPLETED
2016-05-17 22:18:30 Completed Dag: dag_1463523510462_0001_1
2016-05-17 22:18:30,917 INFO app.DAGAppMaster (DAGAppMaster.java:handle(579)) - DAG completed, dagId=dag_1463523510462_0001_1, dagState=SUCCEEDED
2016-05-17 22:18:30,917 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(164)) - Redirecting log file based on addend: dag_1463523510462_0001_1_post
2016-05-17 22:18:30,917 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(179)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-17 22:18:30,917 INFO app.DAGAppMaster (DAGAppMaster.java:handle(625)) - Waiting for next DAG to be submitted.
2016-05-17 22:18:30,925 INFO client.TezClient (TezClient.java:stop(495)) - Shutting down Tez Session, sessionName=(1/1) ...m/testSimpleBuffer/simple, applicationId=application_1463523510462_0001
2016-05-17 22:18:30,928 INFO client.DAGClientHandler (DAGClientHandler.java:shutdownAM(117)) - Received message to shutdown AM
2016-05-17 22:18:30,928 INFO rm.TaskSchedulerEventHandler (TaskSchedulerEventHandler.java:setShouldUnregisterFlag(580)) - TaskScheduler notified that it should unregister from RM
2016-05-17 22:18:30,928 INFO app.DAGAppMaster (DAGAppMaster.java:shutdownTezAM(1088)) - No current running DAG, shutting down the AM
2016-05-17 22:18:30,928 INFO app.DAGAppMaster (DAGAppMaster.java:shutdown(692)) - DAGAppMasterShutdownHandler invoked
2016-05-17 22:18:30,929 INFO app.DAGAppMaster (DAGAppMaster.java:shutdown(701)) - Handling DAGAppMaster shutdown
2016-05-17 22:18:30,929 INFO app.DAGAppMaster (DAGAppMaster.java:run(721)) - Sleeping for 5 seconds before shutting down
2016-05-17 22:18:30,930 INFO history.HistoryEventHandler (HistoryEventHandler.java:serviceStop(79)) - Stopping HistoryEventHandler
2016-05-17 22:18:30,931 INFO recovery.RecoveryService (RecoveryService.java:serviceStop(189)) - Stopping RecoveryService
2016-05-17 22:18:30,931 INFO recovery.RecoveryService (RecoveryService.java:serviceStop(192)) - Handle the remaining events in queue, queue size=0
2016-05-17 22:18:30,931 INFO recovery.RecoveryService (RecoveryService.java:run(166)) - EventQueue take interrupted. Returning
2016-05-17 22:18:30,932 INFO recovery.RecoveryService (RecoveryService.java:serviceStop(214)) - Closing Summary Stream
2016-05-17 22:18:30,933 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:serviceStop(118)) - Stopping SimpleHistoryLoggingService, eventQueueBacklog=0
2016-05-17 22:18:30,934 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:run(105)) - EventQueue take interrupted. Returning
2016-05-17 22:18:30,941 INFO app.DAGAppMaster (DAGAppMaster.java:run(1773)) - Completed deletion of tez scratch data dir, path=file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.5.x-tez-0.6.x/build/test/tmp/.staging/7D2FB5646B554769B5E79B37A91DAD15/FE1AD6A61E6A4290B0EA5055EE3472AA/.tez/application_1463523510462_0001
2016-05-17 22:18:30,942 INFO util.Hadoop18TapUtil (Hadoop18TapUtil.java:cleanTempPath(226)) - deleting temp path /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.5.x-tez-0.6.x/build/test/output/hadoop2-tez/bufferpipesplatform/testSimpleBuffer/simple/_temporary
2016-05-17 22:18:30,942 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [test] completed in: 00:00.503
2016-05-17 22:18:31,052 INFO cascading.PlatformTestCase (PlatformTestCase.java:tearDown(189)) - copying to local /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.5.x-tez-0.6.x/build/test/output//hadoop2-tez/bufferpipesplatform/testSimpleBuffer/simple
2016-05-17 22:18:31,094 INFO util.Util (Util.java:findMainClass(1374)) - using default application jar, may cause class not found exceptions on the cluster
2016-05-17 22:18:31,095 INFO planner.Hadoop2TezPlanner (Hadoop2TezPlanner.java:initialize(151)) - using application jar: /opt/concurrent/home/teamcity/.gradle/caches/modules-2/files-2.1/cascading/cascading-hadoop2-tez/3.1.0/226af02e24d5195c60e4ebe911d0c7c9780f59f0/cascading-hadoop2-tez-3.1.0.jar
2016-05-17 22:18:31,174 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [lower*upper] executed rule registry: NoHashJoinHadoop2TezRuleRegistry, completed as: SUCCESS, in: 00:00.040
2016-05-17 22:18:31,187 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [lower*upper] executed rule registry: HashJoinHadoop2TezRuleRegistry, completed as: SUCCESS, in: 00:00.054
2016-05-17 22:18:31,187 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [lower*upper] rule registry: NoHashJoinHadoop2TezRuleRegistry, supports assembly with steps: 1, nodes: 3
2016-05-17 22:18:31,188 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [lower*upper] rule registry: HashJoinHadoop2TezRuleRegistry, supports assembly with steps: 1, nodes: 3
2016-05-17 22:18:31,188 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [lower*upper] rule registry: NoHashJoinHadoop2TezRuleRegistry, result was selected using: 'default comparator: selects plan with fewest steps and fewest nodes'
2016-05-17 22:18:31,213 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [lower*upper] tap forcing step to tez local mode: /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.5.x-tez-0.6.x/build/classes/test/data/upper.txt
2016-05-17 22:18:31,225 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [lower*upper] tap forcing step to tez local mode: /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.5.x-tez-0.6.x/build/classes/test/data/lower.txt
2016-05-17 22:18:31,235 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [lower*upper] tap forcing step to tez local mode: /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.5.x-tez-0.6.x/build/test/output/hadoop2-tez/bufferpipesplatform/testJoinerClosure/cogroup
2016-05-17 22:18:31,369 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [lower*upper] starting
2016-05-17 22:18:31,369 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [lower*upper] source: Hfs["TextLine[['offset', 'line']->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.5.x-tez-0.6.x/build/classes/test/data/upper.txt"]
2016-05-17 22:18:31,370 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [lower*upper] source: Hfs["TextLine[['offset', 'line']->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.5.x-tez-0.6.x/build/classes/test/data/lower.txt"]
2016-05-17 22:18:31,370 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [lower*upper] sink: Hfs["TextLine[['line']->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.5.x-tez-0.6.x/build/test/output/hadoop2-tez/bufferpipesplatform/testJoinerClosure/cogroup"]
2016-05-17 22:18:31,370 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [lower*upper] parallel execution of steps is enabled: false
2016-05-17 22:18:31,371 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [lower*upper] executing total steps: 1
2016-05-17 22:18:31,371 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [lower*upper] allocating management threads: 1
2016-05-17 22:18:31,372 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [lower*upper] starting step: (1/1) ...testJoinerClosure/cogroup
2016-05-17 22:18:31,372 WARN flow.Flow (BaseFlow.java:logWarn(1516)) - [lower*upper] 'yarn.timeline-service.enabled' is disabled, please enable to capture detailed metrics of completed flows, this may require starting the YARN timeline server daemon
2016-05-17 22:18:31,391 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [lower*upper] tez session mode enabled: true
2016-05-17 22:18:31,396 INFO client.TezClient (TezClient.java:<init>(153)) - Tez Client Version: [ component=tez-api, version=0.6.2, revision=c04283bf91fd96823eec37056d5c8b1a92728265, SCM-URL=scm:git:https://git-wip-us.apache.org/repos/asf/tez.git, buildTime=20150804-1436 ]
2016-05-17 22:18:31,396 INFO client.TezClient (TezClient.java:start(333)) - Session mode. Starting session.
2016-05-17 22:18:31,396 INFO client.TezClientUtils (TezClientUtils.java:setupTezJarsLocalResources(159)) - Ignoring 'tez.lib.uris' since 'tez.ignore.lib.uris' is set to true
2016-05-17 22:18:31,405 INFO client.TezClient (TezCommonUtils.java:createTezSystemStagingPath(118)) - Tez system stage directory file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.5.x-tez-0.6.x/build/test/tmp/.staging/66053CB505334C8DB685D7EB09C005D3/D8F904EC7C6D47DFBA81E1EFBF1F555A/.tez/application_1463523511396_0001 doesn't exist and is created
2016-05-17 22:18:31,424 INFO client.LocalClient (LocalClient.java:createDAGAppMaster(328)) - DAGAppMaster thread has been created
2016-05-17 22:18:31,424 INFO client.LocalClient (LocalClient.java:startDAGAppMaster(251)) - DAGAppMaster is not created wait for 100ms...
2016-05-17 22:18:31,424 INFO client.LocalClient (LocalClient.java:run(275)) - Using staging directory: /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.5.x-tez-0.6.x/build/test/tmp/.staging/66053CB505334C8DB685D7EB09C005D3/D8F904EC7C6D47DFBA81E1EFBF1F555A/.tez/application_1463523511396_0001
2016-05-17 22:18:31,428 INFO app.DAGAppMaster (DAGAppMaster.java:<init>(309)) - Created DAGAppMaster for application appattempt_1463523511396_0001_000000, versionInfo=[ component=tez-dag, version=0.6.2, revision=c04283bf91fd96823eec37056d5c8b1a92728265, SCM-URL=scm:git:https://git-wip-us.apache.org/repos/asf/tez.git, buildTime=20150804-1436 ]
2016-05-17 22:18:31,444 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(333)) - Comparing client version with AM version, clientVersion=0.6.2, AMVersion=0.6.2
2016-05-17 22:18:31,445 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(386)) - Adding session token to jobTokenSecretManager for application
2016-05-17 22:18:31,447 INFO node.AMNodeTracker (AMNodeTracker.java:serviceInit(77)) - blacklistDisablePercent is 33, blacklistingEnabled: false, maxTaskFailuresPerNode: 1
2016-05-17 22:18:31,447 INFO history.HistoryEventHandler (HistoryEventHandler.java:serviceInit(50)) - Initializing HistoryEventHandler
2016-05-17 22:18:31,447 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:serviceInit(66)) - Log file location for SimpleHistoryLoggingService not specified, defaulting to containerLogDir=/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.5.x-tez-0.6.x/build/test/tmp/.staging/66053CB505334C8DB685D7EB09C005D3/D8F904EC7C6D47DFBA81E1EFBF1F555A/.tez/application_1463523511396_0001/localmode-log-dir
2016-05-17 22:18:31,448 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:serviceInit(88)) - Initializing SimpleHistoryLoggingService, logFileLocation=/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.5.x-tez-0.6.x/build/test/tmp/.staging/66053CB505334C8DB685D7EB09C005D3/D8F904EC7C6D47DFBA81E1EFBF1F555A/.tez/application_1463523511396_0001/localmode-log-dir/history.txt.appattempt_1463523511396_0001_000000, maxErrors=10
2016-05-17 22:18:31,448 INFO recovery.RecoveryService (RecoveryService.java:serviceInit(109)) - Initializing RecoveryService
2016-05-17 22:18:31,448 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:N/A][Event:AM_LAUNCHED]: appAttemptId=appattempt_1463523511396_0001_000000, appSubmitTime=1463523511428, launchTime=1463523511428
2016-05-17 22:18:31,459 INFO client.DAGClientServer (DAGClientServer.java:serviceStart(91)) - Instantiated DAGClientRPCServer at ip-10-123-148-208.ec2.internal/10.123.148.208:58056
2016-05-17 22:18:31,459 INFO history.HistoryEventHandler (HistoryEventHandler.java:serviceStart(73)) - Starting HistoryEventHandler
2016-05-17 22:18:31,460 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:serviceStart(95)) - Starting SimpleHistoryLoggingService
2016-05-17 22:18:31,461 INFO recovery.RecoveryService (RecoveryService.java:serviceStart(127)) - Starting RecoveryService
2016-05-17 22:18:31,462 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event AM_LAUNCHED to history file
2016-05-17 22:18:31,463 WARN impl.MetricsConfig (MetricsConfig.java:loadFirst(124)) - Cannot locate configuration: tried hadoop-metrics2-dagappmaster.properties,hadoop-metrics2.properties
2016-05-17 22:18:31,466 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:N/A][Event:AM_STARTED]: appAttemptId=appattempt_1463523511396_0001_000000, startTime=1463523511466
2016-05-17 22:18:31,466 INFO app.DAGAppMaster (DAGAppMaster.java:serviceStart(1644)) - In Session mode. Waiting for DAG over RPC
2016-05-17 22:18:31,466 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event AM_STARTED to history file
2016-05-17 22:18:31,524 INFO client.LocalClient (LocalClient.java:startDAGAppMaster(236)) - DAGAppMaster state: IDLE
2016-05-17 22:18:31,524 INFO client.TezClient (TezClient.java:start(367)) - The url to track the Tez Session: N/A
2016-05-17 22:18:31,525 INFO client.TezClient (TezClient.java:submitDAGSession(405)) - Submitting dag to TezSession, sessionName=(1/1) ...testJoinerClosure/cogroup, applicationId=application_1463523511396_0001, dagName=[66053CB505334C8DB685D7EB09C005D3/D8F904EC7C6D47DFBA81E1EFBF1F555A] lower*upper/(1/1) ...testJoinerClosure/cogroup
2016-05-17 22:18:31,570 INFO app.DAGAppMaster (DAGAppMaster.java:submitDAGToAppMaster(1126)) - Starting DAG submitted via RPC: [66053CB505334C8DB685D7EB09C005D3/D8F904EC7C6D47DFBA81E1EFBF1F555A] lower*upper/(1/1) ...testJoinerClosure/cogroup
2016-05-17 22:18:31,572 INFO app.DAGAppMaster (DAGAppMaster.java:generateDAGVizFile(870)) - Generating DAG graphviz file, dagId=dag_1463523511396_0001_1, filePath=/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.5.x-tez-0.6.x/build/test/tmp/.staging/66053CB505334C8DB685D7EB09C005D3/D8F904EC7C6D47DFBA81E1EFBF1F555A/.tez/application_1463523511396_0001/localmode-log-dir/dag_1463523511396_0001_1.dot
2016-05-17 22:18:31,574 INFO app.DAGAppMaster (DAGAppMaster.java:writePBTextFile(887)) - Writing DAG plan to: /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.5.x-tez-0.6.x/build/test/tmp/.staging/66053CB505334C8DB685D7EB09C005D3/D8F904EC7C6D47DFBA81E1EFBF1F555A/.tez/application_1463523511396_0001/localmode-log-dir/dag_1463523511396_0001_1-tez-dag.pb.txt
2016-05-17 22:18:31,601 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(164)) - Redirecting log file based on addend: dag_1463523511396_0001_1
2016-05-17 22:18:31,601 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(179)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-17 22:18:31,601 INFO app.DAGAppMaster (DAGAppMaster.java:startDAG(2051)) - Running DAG: [66053CB505334C8DB685D7EB09C005D3/D8F904EC7C6D47DFBA81E1EFBF1F555A] lower*upper/(1/1) ...testJoinerClosure/cogroup
2016-05-17 22:18:31 Running Dag: dag_1463523511396_0001_1
2016-05-17 22:18:31,615 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1463523511396_0001_1][Event:DAG_SUBMITTED]: dagID=dag_1463523511396_0001_1, submitTime=1463523511571
2016-05-17 22:18:31,615 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event DAG_SUBMITTED to history file
2016-05-17 22:18:31,616 INFO impl.VertexImpl (VertexImpl.java:setAdditionalOutputs(4033)) - setting additional outputs for vertex E15CD4CB404340CA9C0C4E21BE696455
2016-05-17 22:18:31,616 INFO impl.DAGImpl (DAGImpl.java:assignDAGScheduler(1381)) - Using DAG Scheduler: org.apache.tez.dag.app.dag.impl.DAGSchedulerNaturalOrderControlled
2016-05-17 22:18:31,617 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1463523511396_0001_1][Event:DAG_INITIALIZED]: dagID=dag_1463523511396_0001_1, initTime=1463523511615
2016-05-17 22:18:31,617 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event DAG_INITIALIZED to history file
2016-05-17 22:18:31,617 INFO impl.DAGImpl (DAGImpl.java:handle(973)) - dag_1463523511396_0001_1 transitioned from NEW to INITED due to event DAG_INIT
2016-05-17 22:18:31,619 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1463523511396_0001_1][Event:DAG_STARTED]: dagID=dag_1463523511396_0001_1, startTime=1463523511619
2016-05-17 22:18:31,620 INFO impl.DAGImpl (DAGImpl.java:transition(1614)) - Added additional resources : [[]] to classpath
2016-05-17 22:18:31,620 INFO impl.DAGImpl (DAGImpl.java:handle(973)) - dag_1463523511396_0001_1 transitioned from INITED to RUNNING due to event DAG_START
2016-05-17 22:18:31,619 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event DAG_STARTED to history file
2016-05-17 22:18:31,620 INFO client.TezClient (TezClient.java:submitDAGSession(478)) - Submitted dag to TezSession, sessionName=(1/1) ...testJoinerClosure/cogroup, applicationId=application_1463523511396_0001, dagName=[66053CB505334C8DB685D7EB09C005D3/D8F904EC7C6D47DFBA81E1EFBF1F555A] lower*upper/(1/1) ...testJoinerClosure/cogroup
2016-05-17 22:18:31,620 INFO impl.VertexImpl (VertexImpl.java:setupVertex(2068)) - Root Inputs exist for Vertex: B36EAAFFEAE6410589B338EC7841B27C : {DE0DD679A61C4B67A2055DA89C73B1E0={InputName=DE0DD679A61C4B67A2055DA89C73B1E0}, {Descriptor=ClassName=org.apache.tez.mapreduce.input.MRInput, hasPayload=true}, {ControllerDescriptor=ClassName=org.apache.tez.mapreduce.common.MRInputAMSplitGenerator, hasPayload=false}}
2016-05-17 22:18:31,621 INFO impl.VertexImpl (VertexImpl.java:setupVertex(2078)) - Starting root input initializer for input: DE0DD679A61C4B67A2055DA89C73B1E0, with class: [org.apache.tez.mapreduce.common.MRInputAMSplitGenerator]
2016-05-17 22:18:31,621 INFO impl.VertexImpl (VertexImpl.java:assignVertexManager(2191)) - Setting vertexManager to RootInputVertexManager for vertex_1463523511396_0001_1_00 [B36EAAFFEAE6410589B338EC7841B27C]
2016-05-17 22:18:31,622 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(2983)) - Num tasks is -1. Expecting VertexManager/InputInitializers/1-1 split to set #tasks for the vertex vertex_1463523511396_0001_1_00 [B36EAAFFEAE6410589B338EC7841B27C]
2016-05-17 22:18:31,622 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(2987)) - Vertex will initialize from input initializer. vertex_1463523511396_0001_1_00 [B36EAAFFEAE6410589B338EC7841B27C]
2016-05-17 22:18:31,622 INFO impl.VertexImpl (VertexImpl.java:setupInputInitializerManager(3933)) - Vertex will initialize via inputInitializers vertex_1463523511396_0001_1_00 [B36EAAFFEAE6410589B338EC7841B27C]. Starting root input initializers: 1
2016-05-17 22:18:31,623 INFO impl.VertexImpl (VertexImpl.java:handle(1564)) - vertex_1463523511396_0001_1_00 [B36EAAFFEAE6410589B338EC7841B27C] transitioned from NEW to INITIALIZING due to event V_INIT
2016-05-17 22:18:31,623 INFO impl.VertexImpl (VertexImpl.java:setupVertex(2068)) - Root Inputs exist for Vertex: D62C16529B4C47909B52CAE72AF84E87 : {111B90B05A8148E9AF749CC134359062={InputName=111B90B05A8148E9AF749CC134359062}, {Descriptor=ClassName=org.apache.tez.mapreduce.input.MRInput, hasPayload=true}, {ControllerDescriptor=ClassName=org.apache.tez.mapreduce.common.MRInputAMSplitGenerator, hasPayload=false}}
2016-05-17 22:18:31,624 INFO impl.VertexImpl (VertexImpl.java:setupVertex(2078)) - Starting root input initializer for input: 111B90B05A8148E9AF749CC134359062, with class: [org.apache.tez.mapreduce.common.MRInputAMSplitGenerator]
2016-05-17 22:18:31,624 INFO impl.VertexImpl (VertexImpl.java:assignVertexManager(2191)) - Setting vertexManager to RootInputVertexManager for vertex_1463523511396_0001_1_01 [D62C16529B4C47909B52CAE72AF84E87]
2016-05-17 22:18:31,624 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(2983)) - Num tasks is -1. Expecting VertexManager/InputInitializers/1-1 split to set #tasks for the vertex vertex_1463523511396_0001_1_01 [D62C16529B4C47909B52CAE72AF84E87]
2016-05-17 22:18:31,624 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(2987)) - Vertex will initialize from input initializer. vertex_1463523511396_0001_1_01 [D62C16529B4C47909B52CAE72AF84E87]
2016-05-17 22:18:31,624 INFO impl.VertexImpl (VertexImpl.java:setupInputInitializerManager(3933)) - Vertex will initialize via inputInitializers vertex_1463523511396_0001_1_01 [D62C16529B4C47909B52CAE72AF84E87]. Starting root input initializers: 1
2016-05-17 22:18:31,626 INFO dag.RootInputInitializerManager (RootInputInitializerManager.java:run(242)) - Starting InputInitializer for Input: DE0DD679A61C4B67A2055DA89C73B1E0 on vertex vertex_1463523511396_0001_1_00 [B36EAAFFEAE6410589B338EC7841B27C]
2016-05-17 22:18:31,628 INFO impl.VertexImpl (VertexImpl.java:handle(1564)) - vertex_1463523511396_0001_1_01 [D62C16529B4C47909B52CAE72AF84E87] transitioned from NEW to INITIALIZING due to event V_INIT
2016-05-17 22:18:31,628 INFO dag.RootInputInitializerManager (RootInputInitializerManager.java:run(242)) - Starting InputInitializer for Input: 111B90B05A8148E9AF749CC134359062 on vertex vertex_1463523511396_0001_1_01 [D62C16529B4C47909B52CAE72AF84E87]
2016-05-17 22:18:31,628 INFO impl.VertexImpl (VertexImpl.java:assignVertexManager(2203)) - Setting vertexManager to ShuffleVertexManager for vertex_1463523511396_0001_1_02 [E15CD4CB404340CA9C0C4E21BE696455]
2016-05-17 22:18:31,629 INFO common.MRInputAMSplitGenerator (MRInputAMSplitGenerator.java:initialize(92)) - Emitting serialized splits: true
2016-05-17 22:18:31,629 INFO common.MRInputAMSplitGenerator (MRInputAMSplitGenerator.java:initialize(110)) - Input DE0DD679A61C4B67A2055DA89C73B1E0 asking for 1 tasks. Headroom: 672 Task Resource: 1024 waves: 1.7
2016-05-17 22:18:31,630 INFO common.MRInputAMSplitGenerator (MRInputAMSplitGenerator.java:initialize(121)) - Grouping input splits
2016-05-17 22:18:31,631 INFO hadoop.MRInputHelpers (MRInputHelpers.java:generateInputSplitsToMem(294)) - Generating mapred api input splits
2016-05-17 22:18:31,631 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:initialize(696)) - Shuffle Vertex Manager: settings minFrac:0.25 maxFrac:0.75 auto:false desiredTaskIput:104857600 minTasks:1
2016-05-17 22:18:31,632 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(3014)) - Creating 1 tasks for vertex: vertex_1463523511396_0001_1_02 [E15CD4CB404340CA9C0C4E21BE696455]
2016-05-17 22:18:31,632 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(3026)) - Directly initializing vertex: vertex_1463523511396_0001_1_02 [E15CD4CB404340CA9C0C4E21BE696455]
2016-05-17 22:18:31,632 INFO impl.VertexImpl (VertexImpl.java:initializeCommitters(1915)) - Invoking committer inits for vertex, vertexId=vertex_1463523511396_0001_1_02 [E15CD4CB404340CA9C0C4E21BE696455]
2016-05-17 22:18:31,632 INFO impl.VertexImpl (VertexImpl.java:initializeCommitters(1927)) - Instantiating committer for output=42695EBD27C342B7ACC27BC70682EE53, vertexId=vertex_1463523511396_0001_1_02 [E15CD4CB404340CA9C0C4E21BE696455], committerClass=org.apache.tez.mapreduce.committer.MROutputCommitter
2016-05-17 22:18:31,632 INFO impl.VertexImpl (VertexImpl.java:run(1945)) - Invoking committer init for output=42695EBD27C342B7ACC27BC70682EE53, vertexId=vertex_1463523511396_0001_1_02 [E15CD4CB404340CA9C0C4E21BE696455]
2016-05-17 22:18:31,635 INFO committer.MROutputCommitter (MROutputCommitter.java:getOutputCommitter(145)) - OutputCommitter for outputName=42695EBD27C342B7ACC27BC70682EE53, vertexName=E15CD4CB404340CA9C0C4E21BE696455, outputCommitterClass=org.apache.hadoop.mapred.FileOutputCommitter
2016-05-17 22:18:31,636 INFO hadoop.MRInputHelpers (MRInputHelpers.java:generateInputSplitsToMem(300)) - NumSplits: 1, SerializedSize: 522
2016-05-17 22:18:31,636 INFO impl.VertexImpl (VertexImpl.java:run(1949)) - Invoking committer setup for output=42695EBD27C342B7ACC27BC70682EE53, vertexId=vertex_1463523511396_0001_1_02 [E15CD4CB404340CA9C0C4E21BE696455]
2016-05-17 22:18:31,638 INFO dag.RootInputInitializerManager (RootInputInitializerManager.java:onSuccess(272)) - Succeeded InputInitializer for Input: DE0DD679A61C4B67A2055DA89C73B1E0 on vertex vertex_1463523511396_0001_1_00 [B36EAAFFEAE6410589B338EC7841B27C]
2016-05-17 22:18:31,638 INFO common.MRInputAMSplitGenerator (MRInputAMSplitGenerator.java:initialize(92)) - Emitting serialized splits: true
2016-05-17 22:18:31,640 INFO common.MRInputAMSplitGenerator (MRInputAMSplitGenerator.java:initialize(110)) - Input 111B90B05A8148E9AF749CC134359062 asking for 1 tasks. Headroom: 672 Task Resource: 1024 waves: 1.7
2016-05-17 22:18:31,640 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event VERTEX_INITIALIZED to history file
2016-05-17 22:18:31,641 INFO common.MRInputAMSplitGenerator (MRInputAMSplitGenerator.java:initialize(121)) - Grouping input splits
2016-05-17 22:18:31,641 INFO hadoop.MRInputHelpers (MRInputHelpers.java:generateInputSplitsToMem(294)) - Generating mapred api input splits
2016-05-17 22:18:31,641 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1463523511396_0001_1][Event:VERTEX_INITIALIZED]: vertexName=E15CD4CB404340CA9C0C4E21BE696455, vertexId=vertex_1463523511396_0001_1_02, initRequestedTime=1463523511628, initedTime=1463523511640, numTasks=1, processorName=cascading.flow.tez.FlowProcessor, additionalInputsCount=0
2016-05-17 22:18:31,643 INFO impl.VertexImpl (VertexImpl.java:handle(1564)) - vertex_1463523511396_0001_1_02 [E15CD4CB404340CA9C0C4E21BE696455] transitioned from NEW to INITED due to event V_INIT
2016-05-17 22:18:31,643 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [lower*upper] submitted tez dag to app master: application_1463523511396_0001, with dag id: dag_1463523511396_0001_1
2016-05-17 22:18:31,643 INFO impl.VertexImpl (VertexImpl.java:canInitVertex(3195)) - Cannot init vertex: vertex_1463523511396_0001_1_00 [B36EAAFFEAE6410589B338EC7841B27C] numTasks: -1 numUnitializedEdges: 0 numInitializedInputs: 0 initWaitsForRootInitializers: true
2016-05-17 22:18:31,644 INFO impl.VertexImpl (VertexImpl.java:setParallelism(1374)) - Got updated RootInputsSpecs: {DE0DD679A61C4B67A2055DA89C73B1E0=forAllWorkUnits=true, update=[1]}
2016-05-17 22:18:31,645 INFO impl.VertexImpl (VertexImpl.java:setParallelism(1393)) - Vertex vertex_1463523511396_0001_1_00 [B36EAAFFEAE6410589B338EC7841B27C] parallelism set to 1
2016-05-17 22:18:31,645 INFO impl.VertexImpl (VertexImpl.java:canInitVertex(3195)) - Cannot init vertex: vertex_1463523511396_0001_1_00 [B36EAAFFEAE6410589B338EC7841B27C] numTasks: 1 numUnitializedEdges: 0 numInitializedInputs: 0 initWaitsForRootInitializers: true
2016-05-17 22:18:31,646 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1463523511396_0001_1][Event:VERTEX_DATA_MOVEMENT_EVENTS_GENERATED]: vertexId=vertex_1463523511396_0001_1_00, eventCount=1
2016-05-17 22:18:31,648 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1463523511396_0001_1][Event:VERTEX_INITIALIZED]: vertexName=B36EAAFFEAE6410589B338EC7841B27C, vertexId=vertex_1463523511396_0001_1_00, initRequestedTime=1463523511620, initedTime=1463523511647, numTasks=1, processorName=cascading.flow.tez.FlowProcessor, additionalInputsCount=1
2016-05-17 22:18:31,648 INFO impl.VertexImpl (VertexImpl.java:startIfPossible(3061)) - Triggering start event for vertex: vertex_1463523511396_0001_1_00 [B36EAAFFEAE6410589B338EC7841B27C] with distanceFromRoot: 0
2016-05-17 22:18:31,647 INFO hadoop.MRInputHelpers (MRInputHelpers.java:generateInputSplitsToMem(300)) - NumSplits: 1, SerializedSize: 522
2016-05-17 22:18:31,649 INFO impl.VertexImpl (VertexImpl.java:handle(1564)) - vertex_1463523511396_0001_1_00 [B36EAAFFEAE6410589B338EC7841B27C] transitioned from INITIALIZING to INITED due to event V_ROOT_INPUT_INITIALIZED
2016-05-17 22:18:31,649 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event VERTEX_INITIALIZED to history file
2016-05-17 22:18:31,649 INFO dag.RootInputInitializerManager (RootInputInitializerManager.java:onSuccess(272)) - Succeeded InputInitializer for Input: 111B90B05A8148E9AF749CC134359062 on vertex vertex_1463523511396_0001_1_01 [D62C16529B4C47909B52CAE72AF84E87]
2016-05-17 22:18:31,650 INFO impl.ImmediateStartVertexManager (ImmediateStartVertexManager.java:scheduleTasks(99)) - Starting 1 in B36EAAFFEAE6410589B338EC7841B27C
2016-05-17 22:18:31,651 INFO impl.VertexImpl (VertexImpl.java:scheduleTasks(1250)) - Routing pending task events for vertex: vertex_1463523511396_0001_1_00 [B36EAAFFEAE6410589B338EC7841B27C]
2016-05-17 22:18:31,654 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event VERTEX_STARTED to history file
2016-05-17 22:18:31,654 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1463523511396_0001_1][Event:VERTEX_STARTED]: vertexId=vertex_1463523511396_0001_1_00, startRequestedTime=1463523511650, startedTime=1463523511650
2016-05-17 22:18:31,655 INFO impl.VertexImpl (VertexImpl.java:handle(1564)) - vertex_1463523511396_0001_1_00 [B36EAAFFEAE6410589B338EC7841B27C] transitioned from INITED to RUNNING due to event V_START
2016-05-17 22:18:31,656 INFO impl.VertexImpl (VertexImpl.java:canInitVertex(3195)) - Cannot init vertex: vertex_1463523511396_0001_1_01 [D62C16529B4C47909B52CAE72AF84E87] numTasks: -1 numUnitializedEdges: 0 numInitializedInputs: 0 initWaitsForRootInitializers: true
2016-05-17 22:18:31,657 INFO impl.VertexImpl (VertexImpl.java:setParallelism(1374)) - Got updated RootInputsSpecs: {111B90B05A8148E9AF749CC134359062=forAllWorkUnits=true, update=[1]}
2016-05-17 22:18:31,657 INFO impl.VertexImpl (VertexImpl.java:setParallelism(1393)) - Vertex vertex_1463523511396_0001_1_01 [D62C16529B4C47909B52CAE72AF84E87] parallelism set to 1
2016-05-17 22:18:31,657 INFO impl.VertexImpl (VertexImpl.java:canInitVertex(3195)) - Cannot init vertex: vertex_1463523511396_0001_1_01 [D62C16529B4C47909B52CAE72AF84E87] numTasks: 1 numUnitializedEdges: 0 numInitializedInputs: 0 initWaitsForRootInitializers: true
2016-05-17 22:18:31,658 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1463523511396_0001_1][Event:VERTEX_DATA_MOVEMENT_EVENTS_GENERATED]: vertexId=vertex_1463523511396_0001_1_01, eventCount=1
2016-05-17 22:18:31,659 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event VERTEX_INITIALIZED to history file
2016-05-17 22:18:31,660 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1463523511396_0001_1][Event:VERTEX_INITIALIZED]: vertexName=D62C16529B4C47909B52CAE72AF84E87, vertexId=vertex_1463523511396_0001_1_01, initRequestedTime=1463523511623, initedTime=1463523511659, numTasks=1, processorName=cascading.flow.tez.FlowProcessor, additionalInputsCount=1
2016-05-17 22:18:31,660 INFO impl.VertexImpl (VertexImpl.java:startIfPossible(3061)) - Triggering start event for vertex: vertex_1463523511396_0001_1_01 [D62C16529B4C47909B52CAE72AF84E87] with distanceFromRoot: 0
2016-05-17 22:18:31,661 INFO impl.VertexImpl (VertexImpl.java:handle(1564)) - vertex_1463523511396_0001_1_01 [D62C16529B4C47909B52CAE72AF84E87] transitioned from INITIALIZING to INITED due to event V_ROOT_INPUT_INITIALIZED
2016-05-17 22:18:31,662 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event TASK_STARTED to history file
2016-05-17 22:18:31,662 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1463523511396_0001_1][Event:TASK_STARTED]: vertexName=B36EAAFFEAE6410589B338EC7841B27C, taskId=task_1463523511396_0001_1_00_000000, scheduledTime=1463523511662, launchTime=1463523511662
2016-05-17 22:18:31,662 INFO impl.TaskImpl (TaskImpl.java:handle(862)) - task_1463523511396_0001_1_00_000000 Task Transitioned from NEW to SCHEDULED due to event T_SCHEDULE
2016-05-17 22:18:31,663 INFO impl.VertexImpl (VertexImpl.java:transition(3155)) - Source vertex started: vertex_1463523511396_0001_1_00 for vertex: vertex_1463523511396_0001_1_02 [E15CD4CB404340CA9C0C4E21BE696455] numStartedSources: 1 numSources: 2
2016-05-17 22:18:31,663 INFO impl.VertexImpl (VertexImpl.java:transition(3160)) - Cannot start vertex: vertex_1463523511396_0001_1_02 [E15CD4CB404340CA9C0C4E21BE696455] numStartedSources: 1 numSources: 2
2016-05-17 22:18:31,663 INFO impl.ImmediateStartVertexManager (ImmediateStartVertexManager.java:scheduleTasks(99)) - Starting 1 in D62C16529B4C47909B52CAE72AF84E87
2016-05-17 22:18:31,663 INFO impl.VertexImpl (VertexImpl.java:scheduleTasks(1250)) - Routing pending task events for vertex: vertex_1463523511396_0001_1_01 [D62C16529B4C47909B52CAE72AF84E87]
2016-05-17 22:18:31,664 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event VERTEX_STARTED to history file
2016-05-17 22:18:31,664 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1463523511396_0001_1][Event:VERTEX_STARTED]: vertexId=vertex_1463523511396_0001_1_01, startRequestedTime=1463523511663, startedTime=1463523511663
2016-05-17 22:18:31,664 INFO impl.VertexImpl (VertexImpl.java:handle(1564)) - vertex_1463523511396_0001_1_01 [D62C16529B4C47909B52CAE72AF84E87] transitioned from INITED to RUNNING due to event V_START
2016-05-17 22:18:31,665 INFO impl.DAGSchedulerNaturalOrderControlled (DAGSchedulerNaturalOrderControlled.java:scheduleTask(111)) - Scheduled vertex: vertex_1463523511396_0001_1_00 [B36EAAFFEAE6410589B338EC7841B27C]
2016-05-17 22:18:31,665 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event TASK_STARTED to history file
2016-05-17 22:18:31,666 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1463523511396_0001_1][Event:TASK_STARTED]: vertexName=D62C16529B4C47909B52CAE72AF84E87, taskId=task_1463523511396_0001_1_01_000000, scheduledTime=1463523511665, launchTime=1463523511665
2016-05-17 22:18:31,666 INFO impl.TaskImpl (TaskImpl.java:handle(862)) - task_1463523511396_0001_1_01_000000 Task Transitioned from NEW to SCHEDULED due to event T_SCHEDULE
2016-05-17 22:18:31,666 INFO impl.VertexImpl (VertexImpl.java:transition(3155)) - Source vertex started: vertex_1463523511396_0001_1_01 for vertex: vertex_1463523511396_0001_1_02 [E15CD4CB404340CA9C0C4E21BE696455] numStartedSources: 2 numSources: 2
2016-05-17 22:18:31,667 INFO impl.VertexImpl (VertexImpl.java:startIfPossible(3061)) - Triggering start event for vertex: vertex_1463523511396_0001_1_02 [E15CD4CB404340CA9C0C4E21BE696455] with distanceFromRoot: 1
2016-05-17 22:18:31,667 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:transition(1014)) - remoteTaskSpec:DAGName : [66053CB505334C8DB685D7EB09C005D3/D8F904EC7C6D47DFBA81E1EFBF1F555A] lower*upper/(1/1) ...testJoinerClosure/cogroup, VertexName: B36EAAFFEAE6410589B338EC7841B27C, VertexParallelism: 1, TaskAttemptID:attempt_1463523511396_0001_1_00_000000_0, processorName=cascading.flow.tez.FlowProcessor, inputSpecListSize=1, outputSpecListSize=1, inputSpecList=[{{ sourceVertexName=DE0DD679A61C4B67A2055DA89C73B1E0, physicalEdgeCount=1, inputClassName=org.apache.tez.mapreduce.input.MRInput }}, ], outputSpecList=[{{ destinationVertexName=E15CD4CB404340CA9C0C4E21BE696455, physicalEdgeCount=1, outputClassName=org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput }}, ]
2016-05-17 22:18:31,667 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:handle(692)) - attempt_1463523511396_0001_1_00_000000_0 TaskAttempt Transitioned from NEW to START_WAIT due to event TA_SCHEDULE
2016-05-17 22:18:31,668 INFO impl.DAGSchedulerNaturalOrderControlled (DAGSchedulerNaturalOrderControlled.java:scheduleTask(111)) - Scheduled vertex: vertex_1463523511396_0001_1_01 [D62C16529B4C47909B52CAE72AF84E87]
2016-05-17 22:18:31,668 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:handleVertexStateUpdate(720)) - Received configured notification : CONFIGURED for vertex: D62C16529B4C47909B52CAE72AF84E87 in vertex: E15CD4CB404340CA9C0C4E21BE696455
2016-05-17 22:18:31,668 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:handleVertexStateUpdate(720)) - Received configured notification : CONFIGURED for vertex: B36EAAFFEAE6410589B338EC7841B27C in vertex: E15CD4CB404340CA9C0C4E21BE696455
2016-05-17 22:18:31,669 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:onVertexStarted(355)) - OnVertexStarted vertex: E15CD4CB404340CA9C0C4E21BE696455 with 2 source tasks and 1 pending tasks
2016-05-17 22:18:31,669 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event VERTEX_STARTED to history file
2016-05-17 22:18:31,669 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1463523511396_0001_1][Event:VERTEX_STARTED]: vertexId=vertex_1463523511396_0001_1_02, startRequestedTime=1463523511668, startedTime=1463523511668
2016-05-17 22:18:31,670 INFO impl.VertexImpl (VertexImpl.java:handle(1564)) - vertex_1463523511396_0001_1_02 [E15CD4CB404340CA9C0C4E21BE696455] transitioned from INITED to RUNNING due to event V_START
2016-05-17 22:18:31,670 INFO rm.TaskSchedulerEventHandler (TaskSchedulerEventHandler.java:handleEvent(149)) - Processing the event EventType: S_TA_LAUNCH_REQUEST
2016-05-17 22:18:31,670 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:transition(1014)) - remoteTaskSpec:DAGName : [66053CB505334C8DB685D7EB09C005D3/D8F904EC7C6D47DFBA81E1EFBF1F555A] lower*upper/(1/1) ...testJoinerClosure/cogroup, VertexName: D62C16529B4C47909B52CAE72AF84E87, VertexParallelism: 1, TaskAttemptID:attempt_1463523511396_0001_1_01_000000_0, processorName=cascading.flow.tez.FlowProcessor, inputSpecListSize=1, outputSpecListSize=1, inputSpecList=[{{ sourceVertexName=111B90B05A8148E9AF749CC134359062, physicalEdgeCount=1, inputClassName=org.apache.tez.mapreduce.input.MRInput }}, ], outputSpecList=[{{ destinationVertexName=E15CD4CB404340CA9C0C4E21BE696455, physicalEdgeCount=1, outputClassName=org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput }}, ]
2016-05-17 22:18:31,671 INFO node.AMNodeTracker (AMNodeTracker.java:nodeSeen(91)) - Adding new node: 127.0.0.1:0
2016-05-17 22:18:31,671 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:handle(692)) - attempt_1463523511396_0001_1_01_000000_0 TaskAttempt Transitioned from NEW to START_WAIT due to event TA_SCHEDULE
2016-05-17 22:18:31,671 INFO rm.TaskSchedulerEventHandler (TaskSchedulerEventHandler.java:handleEvent(149)) - Processing the event EventType: S_TA_LAUNCH_REQUEST
2016-05-17 22:18:31,672 INFO container.AMContainerHelpers (AMContainerHelpers.java:createCommonContainerLaunchContext(112)) - Adding #1 tokens and #0 secret keys for NM use for launching container
2016-05-17 22:18:31,672 INFO container.AMContainerHelpers (AMContainerHelpers.java:createCommonContainerLaunchContext(122)) - Putting shuffle token in serviceData
2016-05-17 22:18:31,673 INFO container.AMContainerImpl (AMContainerImpl.java:transition(521)) - Sending Launch Request for Container with id: container_1463523511396_0001_00_000001
2016-05-17 22:18:31,673 INFO container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1463523511396_0001_00_000001 transitioned from ALLOCATED to LAUNCHING via event C_LAUNCH_REQUEST
2016-05-17 22:18:31,674 INFO container.AMContainerImpl (AMContainerImpl.java:transition(521)) - Sending Launch Request for Container with id: container_1463523511396_0001_00_000002
2016-05-17 22:18:31,674 INFO container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1463523511396_0001_00_000002 transitioned from ALLOCATED to LAUNCHING via event C_LAUNCH_REQUEST
2016-05-17 22:18:31,676 INFO task.TezChild (TezChild.java:newTezChild(432)) - PID, containerIdentifier: null, container_1463523511396_0001_00_000001
2016-05-17 22:18:31,676 WARN impl.MetricsSystemImpl (MetricsSystemImpl.java:init(145)) - DAGAppMaster metrics system already initialized!
2016-05-17 22:18:31,677 INFO task.TezChild (TezChild.java:newTezChild(432)) - PID, containerIdentifier: null, container_1463523511396_0001_00_000002
2016-05-17 22:18:31,678 WARN impl.MetricsSystemImpl (MetricsSystemImpl.java:init(145)) - DAGAppMaster metrics system already initialized!
2016-05-17 22:18:31,678 INFO container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1463523511396_0001_00_000001 transitioned from LAUNCHING to IDLE via event C_LAUNCHED
2016-05-17 22:18:31,678 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event CONTAINER_LAUNCHED to history file
2016-05-17 22:18:31,678 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event CONTAINER_LAUNCHED to history file
2016-05-17 22:18:31,679 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1463523511396_0001_1][Event:CONTAINER_LAUNCHED]: containerId=container_1463523511396_0001_00_000002, launchTime=1463523511678
2016-05-17 22:18:31,679 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1463523511396_0001_1][Event:CONTAINER_LAUNCHED]: containerId=container_1463523511396_0001_00_000001, launchTime=1463523511677
2016-05-17 22:18:31,679 INFO task.ContainerReporter (ContainerReporter.java:call(55)) - Attempting to fetch new task for container container_1463523511396_0001_00_000002
2016-05-17 22:18:31,679 INFO container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1463523511396_0001_00_000002 transitioned from LAUNCHING to IDLE via event C_LAUNCHED
2016-05-17 22:18:31,680 INFO task.ContainerReporter (ContainerReporter.java:call(55)) - Attempting to fetch new task for container container_1463523511396_0001_00_000001
2016-05-17 22:18:31,681 INFO container.AMContainerImpl (AMContainerImpl.java:transition(827)) - Assigned taskAttempt + [attempt_1463523511396_0001_1_01_000000_0] to container: [container_1463523511396_0001_00_000002]
2016-05-17 22:18:31,681 INFO container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1463523511396_0001_00_000002 transitioned from IDLE to RUNNING via event C_PULL_TA
2016-05-17 22:18:31,681 INFO container.AMContainerImpl (AMContainerImpl.java:transition(827)) - Assigned taskAttempt + [attempt_1463523511396_0001_1_00_000000_0] to container: [container_1463523511396_0001_00_000001]
2016-05-17 22:18:31,682 INFO container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1463523511396_0001_00_000001 transitioned from IDLE to RUNNING via event C_PULL_TA
2016-05-17 22:18:31,682 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:transition(1146)) - TaskAttempt: [attempt_1463523511396_0001_1_01_000000_0] started. Is using containerId: [container_1463523511396_0001_00_000002] on NM: [127.0.0.1:0]
2016-05-17 22:18:31,683 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1463523511396_0001_1][Event:TASK_ATTEMPT_STARTED]: vertexName=D62C16529B4C47909B52CAE72AF84E87, taskAttemptId=attempt_1463523511396_0001_1_01_000000_0, startTime=1463523511681, containerId=container_1463523511396_0001_00_000002, nodeId=127.0.0.1:0, inProgressLogs=127.0.0.1:0/node/containerlogs/container_1463523511396_0001_00_000002/teamcity, completedLogs=
2016-05-17 22:18:31,683 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:handle(692)) - attempt_1463523511396_0001_1_01_000000_0 TaskAttempt Transitioned from START_WAIT to RUNNING due to event TA_STARTED_REMOTELY
2016-05-17 22:18:31,684 INFO impl.TaskImpl (TaskImpl.java:handle(862)) - task_1463523511396_0001_1_01_000000 Task Transitioned from SCHEDULED to RUNNING due to event T_ATTEMPT_LAUNCHED
2016-05-17 22:18:31,682 INFO app.TaskAttemptListenerImpTezDag (TaskAttemptListenerImpTezDag.java:getTask(240)) - Container with id: container_1463523511396_0001_00_000002 given task: attempt_1463523511396_0001_1_01_000000_0
2016-05-17 22:18:31,685 INFO task.ContainerReporter (ContainerReporter.java:call(65)) - Got TaskUpdate for containerId= container_1463523511396_0001_00_000002: 0 ms after starting to poll. TaskInfo: shouldDie: false, currentTaskAttemptId: attempt_1463523511396_0001_1_01_000000_0
2016-05-17 22:18:31,685 INFO app.TaskAttemptListenerImpTezDag (TaskAttemptListenerImpTezDag.java:getTask(240)) - Container with id: container_1463523511396_0001_00_000001 given task: attempt_1463523511396_0001_1_00_000000_0
2016-05-17 22:18:31 Starting to run new task attempt: attempt_1463523511396_0001_1_01_000000_0
2016-05-17 22:18:31,686 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(164)) - Redirecting log file based on addend: attempt_1463523511396_0001_1_01_000000_0
2016-05-17 22:18:31,686 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(179)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-17 22:18:31,686 INFO task.TezChild (TezChild.java:handleNewTaskCredentials(273)) - Refreshing UGI since Credentials have changed
2016-05-17 22:18:31,686 INFO task.ContainerReporter (ContainerReporter.java:call(65)) - Got TaskUpdate for containerId= container_1463523511396_0001_00_000001: 0 ms after starting to poll. TaskInfo: shouldDie: false, currentTaskAttemptId: attempt_1463523511396_0001_1_00_000000_0
2016-05-17 22:18:31,685 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:transition(1146)) - TaskAttempt: [attempt_1463523511396_0001_1_00_000000_0] started. Is using containerId: [container_1463523511396_0001_00_000001] on NM: [127.0.0.1:0]
2016-05-17 22:18:31,687 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1463523511396_0001_1][Event:TASK_ATTEMPT_STARTED]: vertexName=B36EAAFFEAE6410589B338EC7841B27C, taskAttemptId=attempt_1463523511396_0001_1_00_000000_0, startTime=1463523511685, containerId=container_1463523511396_0001_00_000001, nodeId=127.0.0.1:0, inProgressLogs=127.0.0.1:0/node/containerlogs/container_1463523511396_0001_00_000001/teamcity, completedLogs=
2016-05-17 22:18:31,687 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:handle(692)) - attempt_1463523511396_0001_1_00_000000_0 TaskAttempt Transitioned from START_WAIT to RUNNING due to event TA_STARTED_REMOTELY
2016-05-17 22:18:31,686 INFO task.TezChild (TezChild.java:handleNewTaskCredentials(276)) - Credentials : #Tokens=1, #SecretKeys=0
2016-05-17 22:18:31,687 INFO impl.TaskImpl (TaskImpl.java:handle(862)) - task_1463523511396_0001_1_00_000000 Task Transitioned from SCHEDULED to RUNNING due to event T_ATTEMPT_LAUNCHED
2016-05-17 22:18:31,687 INFO task.TezChild (TezChild.java:handleNewTaskLocalResources(302)) - Localizing additional local resources for Task : {}
2016-05-17 22:18:31,688 INFO task.TezChild (TezChild.java:handleNewTaskLocalResources(312)) - Done localizing additional resources
2016-05-17 22:18:31 Starting to run new task attempt: attempt_1463523511396_0001_1_00_000000_0
2016-05-17 22:18:31,688 INFO metrics.TaskCounterUpdater (TaskCounterUpdater.java:initResourceCalculatorPlugin(148)) - Using ResourceCalculatorProcessTree : [ ]
2016-05-17 22:18:31,688 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(164)) - Redirecting log file based on addend: attempt_1463523511396_0001_1_00_000000_0
2016-05-17 22:18:31,688 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(179)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-17 22:18:31,688 INFO task.TezChild (TezChild.java:handleNewTaskCredentials(273)) - Refreshing UGI since Credentials have changed
2016-05-17 22:18:31,688 INFO task.TezChild (TezChild.java:handleNewTaskCredentials(276)) - Credentials : #Tokens=1, #SecretKeys=0
2016-05-17 22:18:31,688 INFO task.TezChild (TezChild.java:handleNewTaskLocalResources(302)) - Localizing additional local resources for Task : {}
2016-05-17 22:18:31,688 INFO task.TezChild (TezChild.java:handleNewTaskLocalResources(312)) - Done localizing additional resources
2016-05-17 22:18:31,689 INFO metrics.TaskCounterUpdater (TaskCounterUpdater.java:initResourceCalculatorPlugin(148)) - Using ResourceCalculatorProcessTree : [ ]
2016-05-17 22:18:31,689 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:<init>(149)) - Initializing LogicalIOProcessorRuntimeTask with TaskSpec: DAGName : [66053CB505334C8DB685D7EB09C005D3/D8F904EC7C6D47DFBA81E1EFBF1F555A] lower*upper/(1/1) ...testJoinerClosure/cogroup, VertexName: B36EAAFFEAE6410589B338EC7841B27C, VertexParallelism: 1, TaskAttemptID:attempt_1463523511396_0001_1_00_000000_0, processorName=cascading.flow.tez.FlowProcessor, inputSpecListSize=1, outputSpecListSize=1, inputSpecList=[{{ sourceVertexName=DE0DD679A61C4B67A2055DA89C73B1E0, physicalEdgeCount=1, inputClassName=org.apache.tez.mapreduce.input.MRInput }}, ], outputSpecList=[{{ destinationVertexName=E15CD4CB404340CA9C0C4E21BE696455, physicalEdgeCount=1, outputClassName=org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput }}, ]
2016-05-17 22:18:31,689 INFO resources.MemoryDistributor (MemoryDistributor.java:<init>(85)) - InitialMemoryDistributor (isEnabled=true) invoked with: numInputs=1, numOutputs=1, JVM.maxFree=704643072
2016-05-17 22:18:31,688 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:<init>(149)) - Initializing LogicalIOProcessorRuntimeTask with TaskSpec: DAGName : [66053CB505334C8DB685D7EB09C005D3/D8F904EC7C6D47DFBA81E1EFBF1F555A] lower*upper/(1/1) ...testJoinerClosure/cogroup, VertexName: D62C16529B4C47909B52CAE72AF84E87, VertexParallelism: 1, TaskAttemptID:attempt_1463523511396_0001_1_01_000000_0, processorName=cascading.flow.tez.FlowProcessor, inputSpecListSize=1, outputSpecListSize=1, inputSpecList=[{{ sourceVertexName=111B90B05A8148E9AF749CC134359062, physicalEdgeCount=1, inputClassName=org.apache.tez.mapreduce.input.MRInput }}, ], outputSpecList=[{{ destinationVertexName=E15CD4CB404340CA9C0C4E21BE696455, physicalEdgeCount=1, outputClassName=org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput }}, ]
2016-05-17 22:18:31,690 INFO resources.MemoryDistributor (MemoryDistributor.java:<init>(85)) - InitialMemoryDistributor (isEnabled=true) invoked with: numInputs=1, numOutputs=1, JVM.maxFree=704643072
2016-05-17 22:18:31,689 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event TASK_ATTEMPT_STARTED to history file
2016-05-17 22:18:31,690 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event TASK_ATTEMPT_STARTED to history file
2016-05-17 22:18:31,691 INFO task.TezTaskRunner (TezTaskRunner.java:run(172)) - Initializing task, taskAttemptId=attempt_1463523511396_0001_1_00_000000_0
2016-05-17 22:18:31,691 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(191)) - Initializing LogicalProcessorIORuntimeTask
2016-05-17 22:18:31,691 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(195)) - Creating processor, processorClassName=cascading.flow.tez.FlowProcessor
2016-05-17 22:18:31,692 INFO task.TezTaskRunner (TezTaskRunner.java:run(172)) - Initializing task, taskAttemptId=attempt_1463523511396_0001_1_01_000000_0
2016-05-17 22:18:31,693 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(191)) - Initializing LogicalProcessorIORuntimeTask
2016-05-17 22:18:31,692 INFO impl.TaskImpl (TaskImpl.java:getTaskAttemptTezEvents(481)) - TaskAttempt:attempt_1463523511396_0001_1_00_000000_0 sent events: (0-1)
2016-05-17 22:18:31,693 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initializeLogicalIOProcessor(488)) - Initializing processor, processorClassName=cascading.flow.tez.FlowProcessor
2016-05-17 22:18:31,694 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(389)) - Initializing Input using InputSpec: { sourceVertexName=DE0DD679A61C4B67A2055DA89C73B1E0, physicalEdgeCount=1, inputClassName=org.apache.tez.mapreduce.input.MRInput }
2016-05-17 22:18:31,693 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(195)) - Creating processor, processorClassName=cascading.flow.tez.FlowProcessor
2016-05-17 22:18:31,693 INFO impl.TaskImpl (TaskImpl.java:getTaskAttemptTezEvents(481)) - TaskAttempt:attempt_1463523511396_0001_1_01_000000_0 sent events: (0-1)
2016-05-17 22:18:31,697 INFO tez.FlowProcessor (FlowProcessor.java:initialize(88)) - cascading version: 3.1.0
2016-05-17 22:18:31,697 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:createInput(537)) - Creating Input
2016-05-17 22:18:31,698 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(397)) - Initializing Input with src edge: DE0DD679A61C4B67A2055DA89C73B1E0
2016-05-17 22:18:31,698 INFO resources.MemoryDistributor$RequestorInfo (MemoryDistributor.java:<init>(233)) - Received request: 0, type: INPUT, componentVertexName: DE0DD679A61C4B67A2055DA89C73B1E0
2016-05-17 22:18:31,699 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(389)) - Initializing Input using InputSpec: { sourceVertexName=111B90B05A8148E9AF749CC134359062, physicalEdgeCount=1, inputClassName=org.apache.tez.mapreduce.input.MRInput }
2016-05-17 22:18:31,700 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:createInput(537)) - Creating Input
2016-05-17 22:18:31,700 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(397)) - Initializing Input with src edge: 111B90B05A8148E9AF749CC134359062
2016-05-17 22:18:31,700 INFO resources.MemoryDistributor$RequestorInfo (MemoryDistributor.java:<init>(233)) - Received request: 0, type: INPUT, componentVertexName: 111B90B05A8148E9AF749CC134359062
2016-05-17 22:18:31,706 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(438)) - Initializing Output using OutputSpec: { destinationVertexName=E15CD4CB404340CA9C0C4E21BE696455, physicalEdgeCount=1, outputClassName=org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput }
2016-05-17 22:18:31,706 INFO input.MRInput (MRInput.java:initialize(440)) - Using New mapreduce API: false, split information via event: true
2016-05-17 22:18:31,706 INFO input.MRInput (MRInput.java:initializeInternal(493)) - Initialzed MRInput: DE0DD679A61C4B67A2055DA89C73B1E0
2016-05-17 22:18:31,706 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(403)) - Initialized Input with src edge: DE0DD679A61C4B67A2055DA89C73B1E0
2016-05-17 22:18:31,705 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(438)) - Initializing Output using OutputSpec: { destinationVertexName=E15CD4CB404340CA9C0C4E21BE696455, physicalEdgeCount=1, outputClassName=org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput }
2016-05-17 22:18:31,701 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initializeLogicalIOProcessor(488)) - Initializing processor, processorClassName=cascading.flow.tez.FlowProcessor
2016-05-17 22:18:31,707 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:createOutput(565)) - Creating Output
2016-05-17 22:18:31,707 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(446)) - Initializing Output with dest edge: E15CD4CB404340CA9C0C4E21BE696455
2016-05-17 22:18:31,710 INFO tez.FlowProcessor (FlowProcessor.java:initialize(88)) - cascading version: 3.1.0
2016-05-17 22:18:31,712 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:createOutput(565)) - Creating Output
2016-05-17 22:18:31,713 INFO impl.ExternalSorter (ExternalSorter.java:getInitialMemoryRequirement(293)) - Requested SortBufferSize (tez.runtime.io.sort.mb): 100
2016-05-17 22:18:31,713 INFO resources.MemoryDistributor$RequestorInfo (MemoryDistributor.java:<init>(233)) - Received request: 104857600, type: OUTPUT, componentVertexName: E15CD4CB404340CA9C0C4E21BE696455
2016-05-17 22:18:31,713 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(446)) - Initializing Output with dest edge: E15CD4CB404340CA9C0C4E21BE696455
2016-05-17 22:18:31,715 INFO tez.FlowProcessor (FlowProcessor.java:initialize(94)) - flow node id: B36EAAFFEAE6410589B338EC7841B27C, ordinal: 1
2016-05-17 22:18:31,715 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: B36EAAFFEAE6410589B338EC7841B27C, mem on start (mb), free: 234, total: 491, max: 672
2016-05-17 22:18:31,716 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initializeLogicalIOProcessor(491)) - Initialized processor, processorClassName=cascading.flow.tez.FlowProcessor
2016-05-17 22:18:31,717 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(220)) - Waiting for 2 initializers to finish
2016-05-17 22:18:31,716 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(452)) - Initialized Output with dest edge: E15CD4CB404340CA9C0C4E21BE696455
2016-05-17 22:18:31,716 INFO input.MRInput (MRInput.java:initialize(440)) - Using New mapreduce API: false, split information via event: true
2016-05-17 22:18:31,717 INFO input.MRInput (MRInput.java:initializeInternal(493)) - Initialzed MRInput: 111B90B05A8148E9AF749CC134359062
2016-05-17 22:18:31,717 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(403)) - Initialized Input with src edge: 111B90B05A8148E9AF749CC134359062
2016-05-17 22:18:31,717 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(220)) - Waiting for 1 initializers to finish
2016-05-17 22:18:31,718 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(233)) - All initializers finished
2016-05-17 22:18:31,719 INFO resources.MemoryDistributor (MemoryDistributor.java:makeInitialAllocations(124)) - Using Allocator class: org.apache.tez.runtime.library.resources.WeightedScalingMemoryDistributor
2016-05-17 22:18:31,719 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:getRequestTypeForClass(197)) - Falling back to RequestType.OTHER for class: org.apache.tez.mapreduce.input.MRInput
2016-05-17 22:18:31,719 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:initialProcessMemoryRequestContext(171)) - ScaleFactor: 1, for type: OTHER
2016-05-17 22:18:31,719 INFO impl.ExternalSorter (ExternalSorter.java:getInitialMemoryRequirement(293)) - Requested SortBufferSize (tez.runtime.io.sort.mb): 100
2016-05-17 22:18:31,720 INFO resources.MemoryDistributor$RequestorInfo (MemoryDistributor.java:<init>(233)) - Received request: 104857600, type: OUTPUT, componentVertexName: E15CD4CB404340CA9C0C4E21BE696455
2016-05-17 22:18:31,720 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:initialProcessMemoryRequestContext(171)) - ScaleFactor: 12, for type: SORTED_OUTPUT
2016-05-17 22:18:31,721 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:computeReservedFraction(262)) - InitialReservationFraction=0.3, AdditionalReservationFractionForIOs=0.03, finalReserveFractionUsed=0.32999999999999996
2016-05-17 22:18:31,722 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(452)) - Initialized Output with dest edge: E15CD4CB404340CA9C0C4E21BE696455
2016-05-17 22:18:31,723 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:assignMemory(126)) - Scaling Requests. NumRequests: 2, numScaledRequests: 13, TotalRequested: 104857600, TotalRequestedScaled: 9.679163076923077E7, TotalJVMHeap: 704643072, TotalAvailable: 472110858, TotalRequested/TotalJVMHeap:0.15
2016-05-17 22:18:31,723 INFO tez.FlowProcessor (FlowProcessor.java:initialize(94)) - flow node id: D62C16529B4C47909B52CAE72AF84E87, ordinal: 0
2016-05-17 22:18:31,724 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: D62C16529B4C47909B52CAE72AF84E87, mem on start (mb), free: 233, total: 491, max: 672
2016-05-17 22:18:31,724 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initializeLogicalIOProcessor(491)) - Initialized processor, processorClassName=cascading.flow.tez.FlowProcessor
2016-05-17 22:18:31,724 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(220)) - Waiting for 2 initializers to finish
2016-05-17 22:18:31,724 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(220)) - Waiting for 1 initializers to finish
2016-05-17 22:18:31,724 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(233)) - All initializers finished
2016-05-17 22:18:31,724 INFO resources.MemoryDistributor (MemoryDistributor.java:makeInitialAllocations(124)) - Using Allocator class: org.apache.tez.runtime.library.resources.WeightedScalingMemoryDistributor
2016-05-17 22:18:31,723 INFO resources.MemoryDistributor (MemoryDistributor.java:makeInitialAllocations(138)) - Informing: INPUT, DE0DD679A61C4B67A2055DA89C73B1E0, org.apache.tez.mapreduce.input.MRInput: requested=0, allocated=0
2016-05-17 22:18:31,725 INFO resources.MemoryDistributor (MemoryDistributor.java:makeInitialAllocations(138)) - Informing: OUTPUT, E15CD4CB404340CA9C0C4E21BE696455, org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput: requested=104857600, allocated=104857600
2016-05-17 22:18:31,725 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(255)) - Starting Inputs/Outputs
2016-05-17 22:18:31,725 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(268)) - Input: DE0DD679A61C4B67A2055DA89C73B1E0 being auto started by the framework. Subsequent instances will not be auto-started
2016-05-17 22:18:31,725 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(289)) - Num IOs determined for AutoStart: 1
2016-05-17 22:18:31,725 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(291)) - Waiting for 1 IOs to start
2016-05-17 22:18:31,725 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:getRequestTypeForClass(197)) - Falling back to RequestType.OTHER for class: org.apache.tez.mapreduce.input.MRInput
2016-05-17 22:18:31,725 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:initialProcessMemoryRequestContext(171)) - ScaleFactor: 1, for type: OTHER
2016-05-17 22:18:31,726 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:initialProcessMemoryRequestContext(171)) - ScaleFactor: 12, for type: SORTED_OUTPUT
2016-05-17 22:18:31,726 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:computeReservedFraction(262)) - InitialReservationFraction=0.3, AdditionalReservationFractionForIOs=0.03, finalReserveFractionUsed=0.32999999999999996
2016-05-17 22:18:31,726 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:assignMemory(126)) - Scaling Requests. NumRequests: 2, numScaledRequests: 13, TotalRequested: 104857600, TotalRequestedScaled: 9.679163076923077E7, TotalJVMHeap: 704643072, TotalAvailable: 472110858, TotalRequested/TotalJVMHeap:0.15
2016-05-17 22:18:31,726 INFO resources.MemoryDistributor (MemoryDistributor.java:makeInitialAllocations(138)) - Informing: INPUT, 111B90B05A8148E9AF749CC134359062, org.apache.tez.mapreduce.input.MRInput: requested=0, allocated=0
2016-05-17 22:18:31,726 INFO resources.MemoryDistributor (MemoryDistributor.java:makeInitialAllocations(138)) - Informing: OUTPUT, E15CD4CB404340CA9C0C4E21BE696455, org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput: requested=104857600, allocated=104857600
2016-05-17 22:18:31,726 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(255)) - Starting Inputs/Outputs
2016-05-17 22:18:31,727 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(419)) - Starting Input with src edge: 111B90B05A8148E9AF749CC134359062
2016-05-17 22:18:31,727 INFO input.MRInput (MRInput.java:start(450)) - MRInput setup to received events1
2016-05-17 22:18:31,727 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(421)) - Started Input with src edge: 111B90B05A8148E9AF749CC134359062
2016-05-17 22:18:31,727 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(268)) - Input: 111B90B05A8148E9AF749CC134359062 being auto started by the framework. Subsequent instances will not be auto-started
2016-05-17 22:18:31,727 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(289)) - Num IOs determined for AutoStart: 1
2016-05-17 22:18:31,727 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(419)) - Starting Input with src edge: DE0DD679A61C4B67A2055DA89C73B1E0
2016-05-17 22:18:31,728 INFO input.MRInput (MRInput.java:start(450)) - MRInput setup to received events1
2016-05-17 22:18:31,728 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(421)) - Started Input with src edge: DE0DD679A61C4B67A2055DA89C73B1E0
2016-05-17 22:18:31,728 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(304)) - AutoStartComplete
2016-05-17 22:18:31,729 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(291)) - Waiting for 1 IOs to start
2016-05-17 22:18:31,729 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(304)) - AutoStartComplete
2016-05-17 22:18:31,729 INFO task.TezTaskRunner (TezTaskRunner.java:run(175)) - Running task, taskAttemptId=attempt_1463523511396_0001_1_00_000000_0
2016-05-17 22:18:31,730 INFO input.MRInput (MRInput.java:initFromEventInternal(619)) - Initializing RecordReader from event
2016-05-17 22:18:31,732 INFO input.MRInput (MRInput.java:initFromEventInternal(630)) - Split Details -> SplitClass: org.apache.hadoop.mapred.split.TezGroupedSplit, OldSplit: org.apache.hadoop.mapred.split.TezGroupedSplit@547f5496
2016-05-17 22:18:31,734 INFO graph.Hadoop2TezStreamGraph (Hadoop2TezStreamGraph.java:buildGraph(151)) - using streamed source: Hfs["TextLine[['offset', 'line']->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.5.x-tez-0.6.x/build/classes/test/data/lower.txt"]
2016-05-17 22:18:31,737 INFO task.TezTaskRunner (TezTaskRunner.java:run(175)) - Running task, taskAttemptId=attempt_1463523511396_0001_1_01_000000_0
2016-05-17 22:18:31,737 INFO lib.MRReaderMapred (MRReaderMapred.java:setIncrementalConfigParams(163)) - Processing split: org.apache.hadoop.mapred.split.TezGroupedSplit@547f5496
2016-05-17 22:18:31,737 INFO input.MRInput (MRInput.java:initFromEventInternal(634)) - Initialized RecordReader from event
2016-05-17 22:18:31,738 INFO input.MRInput (MRInput.java:processSplitEvent(586)) - Notifying on RecordReader Initialized
2016-05-17 22:18:31,738 INFO input.MRInput (MRInput.java:initFromEventInternal(619)) - Initializing RecordReader from event
2016-05-17 22:18:31,738 INFO tez.FlowProcessor (FlowProcessor.java:run(121)) - sourcing from: Hfs["TextLine[['offset', 'line']->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.5.x-tez-0.6.x/build/classes/test/data/lower.txt"] streamed: true, id: DE0DD679A61C4B67A2055DA89C73B1E0
2016-05-17 22:18:31,739 INFO tez.FlowProcessor (FlowProcessor.java:run(124)) - sinking to: CoGroup(lower*upper)[by: lower:[{1}:'num'] upper:[{1}:'num']], id: 89F46DD398BA43169B2B67B05CFB2EBB
2016-05-17 22:18:31,740 INFO graph.Hadoop2TezStreamGraph (Hadoop2TezStreamGraph.java:buildGraph(151)) - using streamed source: Hfs["TextLine[['offset', 'line']->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.5.x-tez-0.6.x/build/classes/test/data/upper.txt"]
2016-05-17 22:18:31,740 INFO element.TezGroupGate (TezGroupGate.java:prepare(98)) - calling OrderedPartitionedKVOutput#start() on: CoGroup(lower*upper)[by: lower:[{1}:'num'] upper:[{1}:'num']] 89F46DD398BA43169B2B67B05CFB2EBB
2016-05-17 22:18:31,740 INFO tez.FlowProcessor (FlowProcessor.java:run(121)) - sourcing from: Hfs["TextLine[['offset', 'line']->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.5.x-tez-0.6.x/build/classes/test/data/upper.txt"] streamed: true, id: 111B90B05A8148E9AF749CC134359062
2016-05-17 22:18:31,740 INFO input.MRInput (MRInput.java:initFromEventInternal(630)) - Split Details -> SplitClass: org.apache.hadoop.mapred.split.TezGroupedSplit, OldSplit: org.apache.hadoop.mapred.split.TezGroupedSplit@4bdc7b2f
2016-05-17 22:18:31,741 INFO tez.FlowProcessor (FlowProcessor.java:run(124)) - sinking to: CoGroup(lower*upper)[by: lower:[{1}:'num'] upper:[{1}:'num']], id: 89F46DD398BA43169B2B67B05CFB2EBB
2016-05-17 22:18:31,741 INFO element.TezGroupGate (TezGroupGate.java:prepare(98)) - calling OrderedPartitionedKVOutput#start() on: CoGroup(lower*upper)[by: lower:[{1}:'num'] upper:[{1}:'num']] 89F46DD398BA43169B2B67B05CFB2EBB
2016-05-17 22:18:31,744 INFO impl.ExternalSorter (ExternalSorter.java:<init>(167)) - keySerializer=cascading.tuple.hadoop.io.KeyTupleSerializer@4d41f8b3; valueSerializer=cascading.tuple.hadoop.io.ValueTupleSerializer@262ed765; comparator=cascading.tuple.hadoop.util.TupleComparator@4d3f3b78; conf=cascading.tuple.hadoop.TupleSerialization,org.apache.hadoop.io.serializer.WritableSerialization,org.apache.hadoop.io.serializer.avro.AvroSpecificSerialization,org.apache.hadoop.io.serializer.avro.AvroReflectSerialization
2016-05-17 22:18:31,744 INFO lib.MRReaderMapred (MRReaderMapred.java:setIncrementalConfigParams(163)) - Processing split: org.apache.hadoop.mapred.split.TezGroupedSplit@4bdc7b2f
2016-05-17 22:18:31,745 INFO input.MRInput (MRInput.java:initFromEventInternal(634)) - Initialized RecordReader from event
2016-05-17 22:18:31,745 INFO input.MRInput (MRInput.java:processSplitEvent(586)) - Notifying on RecordReader Initialized
2016-05-17 22:18:31,745 INFO impl.ExternalSorter (ExternalSorter.java:<init>(224)) - Instantiating Partitioner: [cascading.tuple.tez.util.TuplePartitioner]
2016-05-17 22:18:31,746 INFO common.TezRuntimeUtils (TezRuntimeUtils.java:instantiatePartitioner(109)) - Using partitioner class: cascading.tuple.tez.util.TuplePartitioner
2016-05-17 22:18:31,746 INFO common.TezRuntimeUtils (TezRuntimeUtils.java:instantiateCombiner(65)) - No combiner specified via tez.runtime.combiner.class. Combiner will not be used
2016-05-17 22:18:31,746 INFO impl.ExternalSorter (ExternalSorter.java:<init>(167)) - keySerializer=cascading.tuple.hadoop.io.KeyTupleSerializer@33b44e09; valueSerializer=cascading.tuple.hadoop.io.ValueTupleSerializer@26f4b64; comparator=cascading.tuple.hadoop.util.TupleComparator@47c49dc1; conf=cascading.tuple.hadoop.TupleSerialization,org.apache.hadoop.io.serializer.WritableSerialization,org.apache.hadoop.io.serializer.avro.AvroSpecificSerialization,org.apache.hadoop.io.serializer.avro.AvroReflectSerialization
2016-05-17 22:18:31,759 INFO dflt.DefaultSorter (DefaultSorter.java:setEquator(328)) - (EQUATOR) 0 kvi 26214396(104857584)
2016-05-17 22:18:31,760 INFO dflt.DefaultSorter (DefaultSorter.java:<init>(150)) - tez.runtime.io.sort.mb: 100
2016-05-17 22:18:31,760 INFO dflt.DefaultSorter (DefaultSorter.java:<init>(151)) - soft limit at 83886080
2016-05-17 22:18:31,760 INFO dflt.DefaultSorter (DefaultSorter.java:<init>(152)) - bufstart = 0; bufvoid = 104857600
2016-05-17 22:18:31,760 INFO dflt.DefaultSorter (DefaultSorter.java:<init>(153)) - kvstart = 26214396; length = 6553600
2016-05-17 22:18:31,760 INFO impl.ExternalSorter (ExternalSorter.java:<init>(224)) - Instantiating Partitioner: [cascading.tuple.tez.util.TuplePartitioner]
2016-05-17 22:18:31,761 INFO common.TezRuntimeUtils (TezRuntimeUtils.java:instantiatePartitioner(109)) - Using partitioner class: cascading.tuple.tez.util.TuplePartitioner
2016-05-17 22:18:31,761 INFO element.TezSourceStage (TezSourceStage.java:prepare(63)) - calling MRInput#start() on: Hfs["TextLine[['offset', 'line']->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.5.x-tez-0.6.x/build/classes/test/data/lower.txt"]
2016-05-17 22:18:31,761 INFO input.MRInput (MRInput.java:start(450)) - MRInput setup to received events1
2016-05-17 22:18:31,761 INFO common.TezRuntimeUtils (TezRuntimeUtils.java:instantiateCombiner(65)) - No combiner specified via tez.runtime.combiner.class. Combiner will not be used
2016-05-17 22:18:31,761 INFO tez.FlowProcessor (FlowProcessor.java:waitForInputsReady(202)) - flow node id: B36EAAFFEAE6410589B338EC7841B27C, all 1 inputs ready in: 00:00:00.000
2016-05-17 22:18:32,142 INFO dflt.DefaultSorter (DefaultSorter.java:setEquator(328)) - (EQUATOR) 0 kvi 26214396(104857584)
2016-05-17 22:18:32,142 INFO dflt.DefaultSorter (DefaultSorter.java:<init>(150)) - tez.runtime.io.sort.mb: 100
2016-05-17 22:18:32,142 INFO dflt.DefaultSorter (DefaultSorter.java:<init>(151)) - soft limit at 83886080
2016-05-17 22:18:32,143 INFO dflt.DefaultSorter (DefaultSorter.java:<init>(152)) - bufstart = 0; bufvoid = 104857600
2016-05-17 22:18:32,143 INFO dflt.DefaultSorter (DefaultSorter.java:<init>(153)) - kvstart = 26214396; length = 6553600
2016-05-17 22:18:32,144 INFO element.TezSourceStage (TezSourceStage.java:prepare(63)) - calling MRInput#start() on: Hfs["TextLine[['offset', 'line']->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.5.x-tez-0.6.x/build/classes/test/data/upper.txt"]
2016-05-17 22:18:32,144 INFO input.MRInput (MRInput.java:start(450)) - MRInput setup to received events1
2016-05-17 22:18:32,144 INFO tez.FlowProcessor (FlowProcessor.java:waitForInputsReady(202)) - flow node id: D62C16529B4C47909B52CAE72AF84E87, all 1 inputs ready in: 00:00:00.000
2016-05-17 22:18:32,145 INFO task.TezTaskRunner (TezTaskRunner.java:run(177)) - Closing task, taskAttemptId=attempt_1463523511396_0001_1_00_000000_0
2016-05-17 22:18:32,145 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: B36EAAFFEAE6410589B338EC7841B27C, mem on close (mb), free: 310, total: 528, max: 672
2016-05-17 22:18:32,145 INFO dflt.DefaultSorter (DefaultSorter.java:flush(589)) - Starting flush of map output
2016-05-17 22:18:32,146 INFO dflt.DefaultSorter (DefaultSorter.java:flush(607)) - Sorting & Spilling map output
2016-05-17 22:18:32,146 INFO dflt.DefaultSorter (DefaultSorter.java:flush(608)) - bufstart = 0; bufend = 75; bufvoid = 104857600
2016-05-17 22:18:32,146 INFO dflt.DefaultSorter (DefaultSorter.java:flush(610)) - kvstart = 26214396(104857584); kvend = 26214380(104857520); length = 17/6553600
2016-05-17 22:18:32,148 INFO task.TezTaskRunner (TezTaskRunner.java:run(177)) - Closing task, taskAttemptId=attempt_1463523511396_0001_1_01_000000_0
2016-05-17 22:18:32,148 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: D62C16529B4C47909B52CAE72AF84E87, mem on close (mb), free: 310, total: 528, max: 672
2016-05-17 22:18:32,148 INFO dflt.DefaultSorter (DefaultSorter.java:flush(589)) - Starting flush of map output
2016-05-17 22:18:32,148 INFO dflt.DefaultSorter (DefaultSorter.java:flush(607)) - Sorting & Spilling map output
2016-05-17 22:18:32,148 INFO dflt.DefaultSorter (DefaultSorter.java:flush(608)) - bufstart = 0; bufend = 75; bufvoid = 104857600
2016-05-17 22:18:32,148 INFO dflt.DefaultSorter (DefaultSorter.java:flush(610)) - kvstart = 26214396(104857584); kvend = 26214380(104857520); length = 17/6553600
2016-05-17 22:18:32,152 INFO dflt.DefaultSorter (DefaultSorter.java:spill(829)) - Finished spill 0
2016-05-17 22:18:32,152 INFO dflt.DefaultSorter (DefaultSorter.java:spill(829)) - Finished spill 0
2016-05-17 22:18:32 Completed running task attempt: attempt_1463523511396_0001_1_01_000000_0
2016-05-17 22:18:32 Completed running task attempt: attempt_1463523511396_0001_1_00_000000_0
2016-05-17 22:18:32,158 INFO task.TezTaskRunner (TezTaskRunner.java:run(181)) - Task completed, taskAttemptId=attempt_1463523511396_0001_1_01_000000_0, fatalErrorOccurred=false
2016-05-17 22:18:32,159 INFO task.TezTaskRunner (TezTaskRunner.java:run(181)) - Task completed, taskAttemptId=attempt_1463523511396_0001_1_00_000000_0, fatalErrorOccurred=false
2016-05-17 22:18:32,160 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1463523511396_0001_1][Event:VERTEX_DATA_MOVEMENT_EVENTS_GENERATED]: vertexId=vertex_1463523511396_0001_1_01, eventCount=1
2016-05-17 22:18:32,160 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:cleanup(712)) - Final Counters : Counters: 27 [[File System Counters FILE_BYTES_READ=124, FILE_BYTES_WRITTEN=1294, FILE_READ_OPS=0, FILE_LARGE_READ_OPS=0, FILE_WRITE_OPS=0][org.apache.tez.common.counters.TaskCounter SPILLED_RECORDS=5, GC_TIME_MILLIS=366, CPU_MILLISECONDS=0, PHYSICAL_MEMORY_BYTES=0, VIRTUAL_MEMORY_BYTES=0, COMMITTED_HEAP_BYTES=553648128, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=5, OUTPUT_BYTES=75, OUTPUT_BYTES_WITH_OVERHEAD=91, OUTPUT_BYTES_PHYSICAL=95, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=0, ADDITIONAL_SPILL_COUNT=0][cascading.flow.SliceCounters Process_Begin_Time=1463523512144, Process_Duration=4, Process_End_Time=1463523512148, Read_Duration=3, Tuples_Read=5, Tuples_Written=5, Write_Duration=1][cascading.flow.StepCounters Tuples_Read=5]]
2016-05-17 22:18:32,160 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event TASK_ATTEMPT_FINISHED to history file
2016-05-17 22:18:32,160 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(164)) - Redirecting log file based on addend:
2016-05-17 22:18:32,161 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(179)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-17 22:18:32,161 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1463523511396_0001_1][Event:TASK_ATTEMPT_FINISHED]: vertexName=D62C16529B4C47909B52CAE72AF84E87, taskAttemptId=attempt_1463523511396_0001_1_01_000000_0, startTime=1463523511681, finishTime=1463523512160, timeTaken=479, status=SUCCEEDED, errorEnum=, diagnostics=, counters=Counters: 28, org.apache.tez.common.counters.DAGCounter, RACK_LOCAL_TASKS=1, File System Counters, FILE_BYTES_READ=124, FILE_BYTES_WRITTEN=1294, FILE_READ_OPS=0, FILE_LARGE_READ_OPS=0, FILE_WRITE_OPS=0, org.apache.tez.common.counters.TaskCounter, SPILLED_RECORDS=5, GC_TIME_MILLIS=366, CPU_MILLISECONDS=0, PHYSICAL_MEMORY_BYTES=0, VIRTUAL_MEMORY_BYTES=0, COMMITTED_HEAP_BYTES=553648128, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=5, OUTPUT_BYTES=75, OUTPUT_BYTES_WITH_OVERHEAD=91, OUTPUT_BYTES_PHYSICAL=95, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=0, ADDITIONAL_SPILL_COUNT=0, cascading.flow.SliceCounters, Process_Begin_Time=1463523512144, Process_Duration=4, Process_End_Time=1463523512148, Read_Duration=3, Tuples_Read=5, Tuples_Written=5, Write_Duration=1, cascading.flow.StepCounters, Tuples_Read=5
2016-05-17 22:18:32,161 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:handle(692)) - attempt_1463523511396_0001_1_01_000000_0 TaskAttempt Transitioned from RUNNING to SUCCEEDED due to event TA_DONE
2016-05-17 22:18:32,161 INFO task.ContainerReporter (ContainerReporter.java:call(55)) - Attempting to fetch new task for container container_1463523511396_0001_00_000002
2016-05-17 22:18:32,162 INFO impl.TaskImpl (TaskImpl.java:transition(1052)) - Task succeeded with attempt attempt_1463523511396_0001_1_01_000000_0
2016-05-17 22:18:32,162 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event TASK_FINISHED to history file
2016-05-17 22:18:32,163 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1463523511396_0001_1][Event:TASK_FINISHED]: vertexName=D62C16529B4C47909B52CAE72AF84E87, taskId=task_1463523511396_0001_1_01_000000, startTime=1463523511681, finishTime=1463523512162, timeTaken=481, status=SUCCEEDED, successfulAttemptID=attempt_1463523511396_0001_1_01_000000_0, diagnostics=, counters=Counters: 28, org.apache.tez.common.counters.DAGCounter, RACK_LOCAL_TASKS=1, File System Counters, FILE_BYTES_READ=124, FILE_BYTES_WRITTEN=1294, FILE_READ_OPS=0, FILE_LARGE_READ_OPS=0, FILE_WRITE_OPS=0, org.apache.tez.common.counters.TaskCounter, SPILLED_RECORDS=5, GC_TIME_MILLIS=366, CPU_MILLISECONDS=0, PHYSICAL_MEMORY_BYTES=0, VIRTUAL_MEMORY_BYTES=0, COMMITTED_HEAP_BYTES=553648128, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=5, OUTPUT_BYTES=75, OUTPUT_BYTES_WITH_OVERHEAD=91, OUTPUT_BYTES_PHYSICAL=95, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=0, ADDITIONAL_SPILL_COUNT=0, cascading.flow.SliceCounters, Process_Begin_Time=1463523512144, Process_Duration=4, Process_End_Time=1463523512148, Read_Duration=3, Tuples_Read=5, Tuples_Written=5, Write_Duration=1, cascading.flow.StepCounters, Tuples_Read=5
2016-05-17 22:18:32,163 INFO impl.TaskImpl (TaskImpl.java:handle(862)) - task_1463523511396_0001_1_01_000000 Task Transitioned from RUNNING to SUCCEEDED due to event T_ATTEMPT_SUCCEEDED
2016-05-17 22:18:32,163 INFO impl.VertexImpl (VertexImpl.java:transition(3536)) - Num completed Tasks for vertex_1463523511396_0001_1_01 [D62C16529B4C47909B52CAE72AF84E87] : 1
2016-05-17 22:18:32,163 INFO impl.VertexImpl (VertexImpl.java:checkVertexForCompletion(1674)) - Vertex succeeded: vertex_1463523511396_0001_1_01 [D62C16529B4C47909B52CAE72AF84E87]
2016-05-17 22:18:32,165 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1463523511396_0001_1][Event:VERTEX_FINISHED]: vertexName=D62C16529B4C47909B52CAE72AF84E87, vertexId=vertex_1463523511396_0001_1_01, initRequestedTime=1463523511623, initedTime=1463523511659, startRequestedTime=1463523511663, startedTime=1463523511663, finishTime=1463523512163, timeTaken=500, status=SUCCEEDED, diagnostics=, counters=Counters: 28, org.apache.tez.common.counters.DAGCounter, RACK_LOCAL_TASKS=1, File System Counters, FILE_BYTES_READ=124, FILE_BYTES_WRITTEN=1294, FILE_READ_OPS=0, FILE_LARGE_READ_OPS=0, FILE_WRITE_OPS=0, org.apache.tez.common.counters.TaskCounter, SPILLED_RECORDS=5, GC_TIME_MILLIS=366, CPU_MILLISECONDS=0, PHYSICAL_MEMORY_BYTES=0, VIRTUAL_MEMORY_BYTES=0, COMMITTED_HEAP_BYTES=553648128, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=5, OUTPUT_BYTES=75, OUTPUT_BYTES_WITH_OVERHEAD=91, OUTPUT_BYTES_PHYSICAL=95, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=0, ADDITIONAL_SPILL_COUNT=0, cascading.flow.SliceCounters, Process_Begin_Time=1463523512144, Process_Duration=4, Process_End_Time=1463523512148, Read_Duration=3, Tuples_Read=5, Tuples_Written=5, Write_Duration=1, cascading.flow.StepCounters, Tuples_Read=5, vertexStats=firstTaskStartTime=1463523511681, firstTasksToStart=[ task_1463523511396_0001_1_01_000000 ], lastTaskFinishTime=1463523512160, lastTasksToFinish=[ task_1463523511396_0001_1_01_000000 ], minTaskDuration=479, maxTaskDuration=479, avgTaskDuration=479.0, numSuccessfulTasks=1, shortestDurationTasks=[ task_1463523511396_0001_1_01_000000 ], longestDurationTasks=[ task_1463523511396_0001_1_01_000000 ], vertexTaskStats={numFailedTasks=0, numSucceededTasks=1, numKilledTaskAttempts=0, numKilledTasks=0, numFailedTaskAttempts=0, numCompletedTasks=1}
2016-05-17 22:18:32,165 INFO impl.VertexImpl (VertexImpl.java:handle(1564)) - vertex_1463523511396_0001_1_01 [D62C16529B4C47909B52CAE72AF84E87] transitioned from RUNNING to SUCCEEDED due to event V_TASK_COMPLETED
2016-05-17 22:18:32,165 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event VERTEX_FINISHED to history file
2016-05-17 22:18:32,166 INFO impl.VertexImpl (VertexImpl.java:transition(3477)) - Source task attempt completed for vertex: vertex_1463523511396_0001_1_02 [E15CD4CB404340CA9C0C4E21BE696455] attempt: attempt_1463523511396_0001_1_01_000000_0 with state: SUCCEEDED vertexState: RUNNING
2016-05-17 22:18:32,167 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1463523511396_0001_1][Event:VERTEX_DATA_MOVEMENT_EVENTS_GENERATED]: vertexId=vertex_1463523511396_0001_1_00, eventCount=1
2016-05-17 22:18:32,167 INFO impl.DAGImpl (DAGImpl.java:transition(1747)) - Vertex vertex_1463523511396_0001_1_01 [D62C16529B4C47909B52CAE72AF84E87] completed., numCompletedVertices=1, numSuccessfulVertices=1, numFailedVertices=0, numKilledVertices=0, numVertices=3
2016-05-17 22:18:32,167 INFO rm.TaskSchedulerEventHandler (TaskSchedulerEventHandler.java:handleEvent(149)) - Processing the event EventType: S_TA_ENDED
2016-05-17 22:18:32,168 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:cleanup(712)) - Final Counters : Counters: 27 [[File System Counters FILE_BYTES_READ=124, FILE_BYTES_WRITTEN=1294, FILE_READ_OPS=0, FILE_LARGE_READ_OPS=0, FILE_WRITE_OPS=0][org.apache.tez.common.counters.TaskCounter SPILLED_RECORDS=5, GC_TIME_MILLIS=366, CPU_MILLISECONDS=0, PHYSICAL_MEMORY_BYTES=0, VIRTUAL_MEMORY_BYTES=0, COMMITTED_HEAP_BYTES=553648128, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=5, OUTPUT_BYTES=75, OUTPUT_BYTES_WITH_OVERHEAD=91, OUTPUT_BYTES_PHYSICAL=95, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=0, ADDITIONAL_SPILL_COUNT=0][cascading.flow.SliceCounters Process_Begin_Time=1463523512129, Process_Duration=16, Process_End_Time=1463523512145, Read_Duration=15, Tuples_Read=5, Tuples_Written=5, Write_Duration=1][cascading.flow.StepCounters Tuples_Read=5]]
2016-05-17 22:18:32,169 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(164)) - Redirecting log file based on addend:
2016-05-17 22:18:32,169 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(179)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-17 22:18:32,169 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1463523511396_0001_1][Event:TASK_ATTEMPT_FINISHED]: vertexName=B36EAAFFEAE6410589B338EC7841B27C, taskAttemptId=attempt_1463523511396_0001_1_00_000000_0, startTime=1463523511685, finishTime=1463523512167, timeTaken=482, status=SUCCEEDED, errorEnum=, diagnostics=, counters=Counters: 28, org.apache.tez.common.counters.DAGCounter, RACK_LOCAL_TASKS=1, File System Counters, FILE_BYTES_READ=124, FILE_BYTES_WRITTEN=1294, FILE_READ_OPS=0, FILE_LARGE_READ_OPS=0, FILE_WRITE_OPS=0, org.apache.tez.common.counters.TaskCounter, SPILLED_RECORDS=5, GC_TIME_MILLIS=366, CPU_MILLISECONDS=0, PHYSICAL_MEMORY_BYTES=0, VIRTUAL_MEMORY_BYTES=0, COMMITTED_HEAP_BYTES=553648128, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=5, OUTPUT_BYTES=75, OUTPUT_BYTES_WITH_OVERHEAD=91, OUTPUT_BYTES_PHYSICAL=95, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=0, ADDITIONAL_SPILL_COUNT=0, cascading.flow.SliceCounters, Process_Begin_Time=1463523512129, Process_Duration=16, Process_End_Time=1463523512145, Read_Duration=15, Tuples_Read=5, Tuples_Written=5, Write_Duration=1, cascading.flow.StepCounters, Tuples_Read=5
2016-05-17 22:18:32,169 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:handle(692)) - attempt_1463523511396_0001_1_00_000000_0 TaskAttempt Transitioned from RUNNING to SUCCEEDED due to event TA_DONE
2016-05-17 22:18:32,169 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event TASK_ATTEMPT_FINISHED to history file
2016-05-17 22:18:32,172 INFO task.ContainerReporter (ContainerReporter.java:call(55)) - Attempting to fetch new task for container container_1463523511396_0001_00_000001
2016-05-17 22:18:32,172 INFO container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1463523511396_0001_00_000002 transitioned from RUNNING to IDLE via event C_TA_SUCCEEDED
2016-05-17 22:18:32,173 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1463523511396_0001_1][Event:CONTAINER_STOPPED]: containerId=container_1463523511396_0001_00_000002, stoppedTime=1463523512173, exitStatus=0
2016-05-17 22:18:32,173 INFO container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1463523511396_0001_00_000002 transitioned from IDLE to STOP_REQUESTED via event C_STOP_REQUEST
2016-05-17 22:18:32,173 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event CONTAINER_STOPPED to history file
2016-05-17 22:18:32,174 INFO rm.TaskSchedulerEventHandler (TaskSchedulerEventHandler.java:handleEvent(149)) - Processing the event EventType: S_TA_ENDED
2016-05-17 22:18:32,174 INFO impl.TaskImpl (TaskImpl.java:transition(1052)) - Task succeeded with attempt attempt_1463523511396_0001_1_00_000000_0
2016-05-17 22:18:32,174 INFO app.TaskAttemptListenerImpTezDag (TaskAttemptListenerImpTezDag.java:getTask(221)) - No more tasks for container with id : container_1463523511396_0001_00_000002. Asking it to die
2016-05-17 22:18:32,174 INFO task.ContainerReporter (ContainerReporter.java:call(65)) - Got TaskUpdate for containerId= container_1463523511396_0001_00_000002: 12 ms after starting to poll. TaskInfo: shouldDie: true
2016-05-17 22:18:32,175 INFO task.TezChild (TezChild.java:run(207)) - ContainerTask returned shouldDie=true for container container_1463523511396_0001_00_000002, Exiting
2016-05-17 22:18:32,175 INFO task.TezChild (TezChild.java:shutdown(343)) - Shutdown invoked for container container_1463523511396_0001_00_000002
2016-05-17 22:18:32,175 INFO task.TezChild (TezChild.java:shutdown(345)) - Shutting down container container_1463523511396_0001_00_000002
2016-05-17 22:18:32,176 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event TASK_FINISHED to history file
2016-05-17 22:18:32,178 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(270)) - ContainerExecutionResult for: container_1463523511396_0001_00_000002 = ContainerExecutionResult{exitStatus=SUCCESS, throwable=null, errorMessage='Asked to die by the AM'}
2016-05-17 22:18:32,178 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(274)) - Container: container_1463523511396_0001_00_000002 completed successfully
2016-05-17 22:18:32,178 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1463523511396_0001_1][Event:TASK_FINISHED]: vertexName=B36EAAFFEAE6410589B338EC7841B27C, taskId=task_1463523511396_0001_1_00_000000, startTime=1463523511685, finishTime=1463523512174, timeTaken=489, status=SUCCEEDED, successfulAttemptID=attempt_1463523511396_0001_1_00_000000_0, diagnostics=, counters=Counters: 28, org.apache.tez.common.counters.DAGCounter, RACK_LOCAL_TASKS=1, File System Counters, FILE_BYTES_READ=124, FILE_BYTES_WRITTEN=1294, FILE_READ_OPS=0, FILE_LARGE_READ_OPS=0, FILE_WRITE_OPS=0, org.apache.tez.common.counters.TaskCounter, SPILLED_RECORDS=5, GC_TIME_MILLIS=366, CPU_MILLISECONDS=0, PHYSICAL_MEMORY_BYTES=0, VIRTUAL_MEMORY_BYTES=0, COMMITTED_HEAP_BYTES=553648128, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=5, OUTPUT_BYTES=75, OUTPUT_BYTES_WITH_OVERHEAD=91, OUTPUT_BYTES_PHYSICAL=95, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=0, ADDITIONAL_SPILL_COUNT=0, cascading.flow.SliceCounters, Process_Begin_Time=1463523512129, Process_Duration=16, Process_End_Time=1463523512145, Read_Duration=15, Tuples_Read=5, Tuples_Written=5, Write_Duration=1, cascading.flow.StepCounters, Tuples_Read=5
2016-05-17 22:18:32,179 INFO impl.TaskImpl (TaskImpl.java:handle(862)) - task_1463523511396_0001_1_00_000000 Task Transitioned from RUNNING to SUCCEEDED due to event T_ATTEMPT_SUCCEEDED
2016-05-17 22:18:32,179 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:stop(242)) - Ignoring stop request for containerId: container_1463523511396_0001_00_000002
2016-05-17 22:18:32,180 INFO container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1463523511396_0001_00_000001 transitioned from RUNNING to IDLE via event C_TA_SUCCEEDED
2016-05-17 22:18:32,180 INFO impl.VertexImpl (VertexImpl.java:transition(3536)) - Num completed Tasks for vertex_1463523511396_0001_1_00 [B36EAAFFEAE6410589B338EC7841B27C] : 1
2016-05-17 22:18:32,180 INFO impl.VertexImpl (VertexImpl.java:checkVertexForCompletion(1674)) - Vertex succeeded: vertex_1463523511396_0001_1_00 [B36EAAFFEAE6410589B338EC7841B27C]
2016-05-17 22:18:32,181 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event VERTEX_FINISHED to history file
2016-05-17 22:18:32,182 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1463523511396_0001_1][Event:VERTEX_FINISHED]: vertexName=B36EAAFFEAE6410589B338EC7841B27C, vertexId=vertex_1463523511396_0001_1_00, initRequestedTime=1463523511620, initedTime=1463523511647, startRequestedTime=1463523511650, startedTime=1463523511650, finishTime=1463523512180, timeTaken=530, status=SUCCEEDED, diagnostics=, counters=Counters: 28, org.apache.tez.common.counters.DAGCounter, RACK_LOCAL_TASKS=1, File System Counters, FILE_BYTES_READ=124, FILE_BYTES_WRITTEN=1294, FILE_READ_OPS=0, FILE_LARGE_READ_OPS=0, FILE_WRITE_OPS=0, org.apache.tez.common.counters.TaskCounter, SPILLED_RECORDS=5, GC_TIME_MILLIS=366, CPU_MILLISECONDS=0, PHYSICAL_MEMORY_BYTES=0, VIRTUAL_MEMORY_BYTES=0, COMMITTED_HEAP_BYTES=553648128, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=5, OUTPUT_BYTES=75, OUTPUT_BYTES_WITH_OVERHEAD=91, OUTPUT_BYTES_PHYSICAL=95, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=0, ADDITIONAL_SPILL_COUNT=0, cascading.flow.SliceCounters, Process_Begin_Time=1463523512129, Process_Duration=16, Process_End_Time=1463523512145, Read_Duration=15, Tuples_Read=5, Tuples_Written=5, Write_Duration=1, cascading.flow.StepCounters, Tuples_Read=5, vertexStats=firstTaskStartTime=1463523511685, firstTasksToStart=[ task_1463523511396_0001_1_00_000000 ], lastTaskFinishTime=1463523512167, lastTasksToFinish=[ task_1463523511396_0001_1_00_000000 ], minTaskDuration=482, maxTaskDuration=482, avgTaskDuration=482.0, numSuccessfulTasks=1, shortestDurationTasks=[ task_1463523511396_0001_1_00_000000 ], longestDurationTasks=[ task_1463523511396_0001_1_00_000000 ], vertexTaskStats={numFailedTasks=0, numSucceededTasks=1, numKilledTaskAttempts=0, numKilledTasks=0, numFailedTaskAttempts=0, numCompletedTasks=1}
2016-05-17 22:18:32,182 INFO impl.VertexImpl (VertexImpl.java:handle(1564)) - vertex_1463523511396_0001_1_00 [B36EAAFFEAE6410589B338EC7841B27C] transitioned from RUNNING to SUCCEEDED due to event V_TASK_COMPLETED
2016-05-17 22:18:32,182 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1463523511396_0001_1][Event:CONTAINER_STOPPED]: containerId=container_1463523511396_0001_00_000001, stoppedTime=1463523512182, exitStatus=0
2016-05-17 22:18:32,184 INFO container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1463523511396_0001_00_000001 transitioned from IDLE to STOP_REQUESTED via event C_STOP_REQUEST
2016-05-17 22:18:32,184 INFO app.TaskAttemptListenerImpTezDag (TaskAttemptListenerImpTezDag.java:getTask(213)) - Container with id: container_1463523511396_0001_00_000001 is valid, but no longer registered, and will be killed
2016-05-17 22:18:32,185 INFO task.ContainerReporter (ContainerReporter.java:call(65)) - Got TaskUpdate for containerId= container_1463523511396_0001_00_000001: 12 ms after starting to poll. TaskInfo: shouldDie: true
2016-05-17 22:18:32,185 INFO task.TezChild (TezChild.java:run(207)) - ContainerTask returned shouldDie=true for container container_1463523511396_0001_00_000001, Exiting
2016-05-17 22:18:32,185 INFO task.TezChild (TezChild.java:shutdown(343)) - Shutdown invoked for container container_1463523511396_0001_00_000001
2016-05-17 22:18:32,185 INFO task.TezChild (TezChild.java:shutdown(345)) - Shutting down container container_1463523511396_0001_00_000001
2016-05-17 22:18:32,185 INFO container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1463523511396_0001_00_000002 transitioned from STOP_REQUESTED to COMPLETED via event C_COMPLETED
2016-05-17 22:18:32,182 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event CONTAINER_STOPPED to history file
2016-05-17 22:18:32,186 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(270)) - ContainerExecutionResult for: container_1463523511396_0001_00_000001 = ContainerExecutionResult{exitStatus=SUCCESS, throwable=null, errorMessage='Asked to die by the AM'}
2016-05-17 22:18:32,186 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(274)) - Container: container_1463523511396_0001_00_000001 completed successfully
2016-05-17 22:18:32,186 INFO impl.VertexImpl (VertexImpl.java:transition(3477)) - Source task attempt completed for vertex: vertex_1463523511396_0001_1_02 [E15CD4CB404340CA9C0C4E21BE696455] attempt: attempt_1463523511396_0001_1_00_000000_0 with state: SUCCEEDED vertexState: RUNNING
2016-05-17 22:18:32,186 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:schedulePendingTasks(608)) - All source tasks assigned. Ramping up 1 remaining tasks for vertex: E15CD4CB404340CA9C0C4E21BE696455
2016-05-17 22:18:32,186 INFO impl.VertexImpl (VertexImpl.java:scheduleTasks(1250)) - Routing pending task events for vertex: vertex_1463523511396_0001_1_02 [E15CD4CB404340CA9C0C4E21BE696455]
2016-05-17 22:18:32,187 INFO impl.DAGImpl (DAGImpl.java:transition(1747)) - Vertex vertex_1463523511396_0001_1_00 [B36EAAFFEAE6410589B338EC7841B27C] completed., numCompletedVertices=2, numSuccessfulVertices=2, numFailedVertices=0, numKilledVertices=0, numVertices=3
2016-05-17 22:18:32,187 INFO container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1463523511396_0001_00_000001 transitioned from STOP_REQUESTED to COMPLETED via event C_COMPLETED
2016-05-17 22:18:32,187 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:stop(242)) - Ignoring stop request for containerId: container_1463523511396_0001_00_000001
2016-05-17 22:18:32,187 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event TASK_STARTED to history file
2016-05-17 22:18:32,187 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1463523511396_0001_1][Event:TASK_STARTED]: vertexName=E15CD4CB404340CA9C0C4E21BE696455, taskId=task_1463523511396_0001_1_02_000000, scheduledTime=1463523512187, launchTime=1463523512187
2016-05-17 22:18:32,188 INFO impl.TaskImpl (TaskImpl.java:handle(862)) - task_1463523511396_0001_1_02_000000 Task Transitioned from NEW to SCHEDULED due to event T_SCHEDULE
2016-05-17 22:18:32,188 INFO impl.DAGSchedulerNaturalOrderControlled (DAGSchedulerNaturalOrderControlled.java:scheduleTask(111)) - Scheduled vertex: vertex_1463523511396_0001_1_02 [E15CD4CB404340CA9C0C4E21BE696455]
2016-05-17 22:18:32,188 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:transition(1014)) - remoteTaskSpec:DAGName : [66053CB505334C8DB685D7EB09C005D3/D8F904EC7C6D47DFBA81E1EFBF1F555A] lower*upper/(1/1) ...testJoinerClosure/cogroup, VertexName: E15CD4CB404340CA9C0C4E21BE696455, VertexParallelism: 1, TaskAttemptID:attempt_1463523511396_0001_1_02_000000_0, processorName=cascading.flow.tez.FlowProcessor, inputSpecListSize=2, outputSpecListSize=1, inputSpecList=[{{ sourceVertexName=D62C16529B4C47909B52CAE72AF84E87, physicalEdgeCount=1, inputClassName=org.apache.tez.runtime.library.input.OrderedGroupedKVInput }}, {{ sourceVertexName=B36EAAFFEAE6410589B338EC7841B27C, physicalEdgeCount=1, inputClassName=org.apache.tez.runtime.library.input.OrderedGroupedKVInput }}, ], outputSpecList=[{{ destinationVertexName=42695EBD27C342B7ACC27BC70682EE53, physicalEdgeCount=0, outputClassName=org.apache.tez.mapreduce.output.MROutput }}, ]
2016-05-17 22:18:32,188 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:handle(692)) - attempt_1463523511396_0001_1_02_000000_0 TaskAttempt Transitioned from NEW to START_WAIT due to event TA_SCHEDULE
2016-05-17 22:18:32,188 INFO rm.TaskSchedulerEventHandler (TaskSchedulerEventHandler.java:handleEvent(149)) - Processing the event EventType: S_TA_LAUNCH_REQUEST
2016-05-17 22:18:32,189 INFO container.AMContainerImpl (AMContainerImpl.java:transition(521)) - Sending Launch Request for Container with id: container_1463523511396_0001_00_000003
2016-05-17 22:18:32,190 INFO container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1463523511396_0001_00_000003 transitioned from ALLOCATED to LAUNCHING via event C_LAUNCH_REQUEST
2016-05-17 22:18:32,192 INFO task.TezChild (TezChild.java:newTezChild(432)) - PID, containerIdentifier: null, container_1463523511396_0001_00_000003
2016-05-17 22:18:32,192 WARN impl.MetricsConfig (MetricsConfig.java:loadFirst(124)) - Cannot locate configuration: tried hadoop-metrics2-teztask.properties,hadoop-metrics2.properties
2016-05-17 22:18:32,195 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event CONTAINER_LAUNCHED to history file
2016-05-17 22:18:32,196 INFO container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1463523511396_0001_00_000003 transitioned from LAUNCHING to IDLE via event C_LAUNCHED
2016-05-17 22:18:32,196 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1463523511396_0001_1][Event:CONTAINER_LAUNCHED]: containerId=container_1463523511396_0001_00_000003, launchTime=1463523512195
2016-05-17 22:18:32,197 INFO task.ContainerReporter (ContainerReporter.java:call(55)) - Attempting to fetch new task for container container_1463523511396_0001_00_000003
2016-05-17 22:18:32,198 INFO container.AMContainerImpl (AMContainerImpl.java:transition(827)) - Assigned taskAttempt + [attempt_1463523511396_0001_1_02_000000_0] to container: [container_1463523511396_0001_00_000003]
2016-05-17 22:18:32,198 INFO container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1463523511396_0001_00_000003 transitioned from IDLE to RUNNING via event C_PULL_TA
2016-05-17 22:18:32,198 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:transition(1146)) - TaskAttempt: [attempt_1463523511396_0001_1_02_000000_0] started. Is using containerId: [container_1463523511396_0001_00_000003] on NM: [127.0.0.1:0]
2016-05-17 22:18:32,199 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1463523511396_0001_1][Event:TASK_ATTEMPT_STARTED]: vertexName=E15CD4CB404340CA9C0C4E21BE696455, taskAttemptId=attempt_1463523511396_0001_1_02_000000_0, startTime=1463523512198, containerId=container_1463523511396_0001_00_000003, nodeId=127.0.0.1:0, inProgressLogs=127.0.0.1:0/node/containerlogs/container_1463523511396_0001_00_000003/teamcity, completedLogs=
2016-05-17 22:18:32,199 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:handle(692)) - attempt_1463523511396_0001_1_02_000000_0 TaskAttempt Transitioned from START_WAIT to RUNNING due to event TA_STARTED_REMOTELY
2016-05-17 22:18:32,199 INFO impl.TaskImpl (TaskImpl.java:handle(862)) - task_1463523511396_0001_1_02_000000 Task Transitioned from SCHEDULED to RUNNING due to event T_ATTEMPT_LAUNCHED
2016-05-17 22:18:32,199 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event TASK_ATTEMPT_STARTED to history file
2016-05-17 22:18:32,200 INFO app.TaskAttemptListenerImpTezDag (TaskAttemptListenerImpTezDag.java:getTask(240)) - Container with id: container_1463523511396_0001_00_000003 given task: attempt_1463523511396_0001_1_02_000000_0
2016-05-17 22:18:32,200 INFO task.ContainerReporter (ContainerReporter.java:call(65)) - Got TaskUpdate for containerId= container_1463523511396_0001_00_000003: 0 ms after starting to poll. TaskInfo: shouldDie: false, currentTaskAttemptId: attempt_1463523511396_0001_1_02_000000_0
2016-05-17 22:18:32 Starting to run new task attempt: attempt_1463523511396_0001_1_02_000000_0
2016-05-17 22:18:32,201 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(164)) - Redirecting log file based on addend: attempt_1463523511396_0001_1_02_000000_0
2016-05-17 22:18:32,201 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(179)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-17 22:18:32,201 INFO task.TezChild (TezChild.java:handleNewTaskCredentials(273)) - Refreshing UGI since Credentials have changed
2016-05-17 22:18:32,202 INFO task.TezChild (TezChild.java:handleNewTaskCredentials(276)) - Credentials : #Tokens=1, #SecretKeys=0
2016-05-17 22:18:32,202 INFO task.TezChild (TezChild.java:handleNewTaskLocalResources(302)) - Localizing additional local resources for Task : {}
2016-05-17 22:18:32,202 INFO task.TezChild (TezChild.java:handleNewTaskLocalResources(312)) - Done localizing additional resources
2016-05-17 22:18:32,202 INFO metrics.TaskCounterUpdater (TaskCounterUpdater.java:initResourceCalculatorPlugin(148)) - Using ResourceCalculatorProcessTree : [ ]
2016-05-17 22:18:32,203 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:<init>(149)) - Initializing LogicalIOProcessorRuntimeTask with TaskSpec: DAGName : [66053CB505334C8DB685D7EB09C005D3/D8F904EC7C6D47DFBA81E1EFBF1F555A] lower*upper/(1/1) ...testJoinerClosure/cogroup, VertexName: E15CD4CB404340CA9C0C4E21BE696455, VertexParallelism: 1, TaskAttemptID:attempt_1463523511396_0001_1_02_000000_0, processorName=cascading.flow.tez.FlowProcessor, inputSpecListSize=2, outputSpecListSize=1, inputSpecList=[{{ sourceVertexName=D62C16529B4C47909B52CAE72AF84E87, physicalEdgeCount=1, inputClassName=org.apache.tez.runtime.library.input.OrderedGroupedKVInput }}, {{ sourceVertexName=B36EAAFFEAE6410589B338EC7841B27C, physicalEdgeCount=1, inputClassName=org.apache.tez.runtime.library.input.OrderedGroupedKVInput }}, ], outputSpecList=[{{ destinationVertexName=42695EBD27C342B7ACC27BC70682EE53, physicalEdgeCount=0, outputClassName=org.apache.tez.mapreduce.output.MROutput }}, ]
2016-05-17 22:18:32,203 INFO resources.MemoryDistributor (MemoryDistributor.java:<init>(85)) - InitialMemoryDistributor (isEnabled=true) invoked with: numInputs=2, numOutputs=1, JVM.maxFree=704643072
2016-05-17 22:18:32,203 INFO task.TezTaskRunner (TezTaskRunner.java:run(172)) - Initializing task, taskAttemptId=attempt_1463523511396_0001_1_02_000000_0
2016-05-17 22:18:32,204 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(191)) - Initializing LogicalProcessorIORuntimeTask
2016-05-17 22:18:32,204 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(195)) - Creating processor, processorClassName=cascading.flow.tez.FlowProcessor
2016-05-17 22:18:32,205 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(389)) - Initializing Input using InputSpec: { sourceVertexName=D62C16529B4C47909B52CAE72AF84E87, physicalEdgeCount=1, inputClassName=org.apache.tez.runtime.library.input.OrderedGroupedKVInput }
2016-05-17 22:18:32,205 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:createInput(537)) - Creating Input
2016-05-17 22:18:32,205 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(397)) - Initializing Input with src edge: D62C16529B4C47909B52CAE72AF84E87
2016-05-17 22:18:32,207 INFO orderedgrouped.MergeManager (MergeManager.java:getInitialMemoryRequirement(310)) - Initial Shuffle Memory Required: 634178752, based on INPUT_BUFFER_factor: 0.9
2016-05-17 22:18:32,207 INFO orderedgrouped.MergeManager (MergeManager.java:getInitialMemoryRequirement(320)) - Initial Memory required for final merged output: 0, using factor: 0.0
2016-05-17 22:18:32,207 INFO resources.MemoryDistributor$RequestorInfo (MemoryDistributor.java:<init>(233)) - Received request: 634178752, type: INPUT, componentVertexName: D62C16529B4C47909B52CAE72AF84E87
2016-05-17 22:18:32,207 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(403)) - Initialized Input with src edge: D62C16529B4C47909B52CAE72AF84E87
2016-05-17 22:18:32,208 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(389)) - Initializing Input using InputSpec: { sourceVertexName=B36EAAFFEAE6410589B338EC7841B27C, physicalEdgeCount=1, inputClassName=org.apache.tez.runtime.library.input.OrderedGroupedKVInput }
2016-05-17 22:18:32,209 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:createInput(537)) - Creating Input
2016-05-17 22:18:32,209 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(397)) - Initializing Input with src edge: B36EAAFFEAE6410589B338EC7841B27C
2016-05-17 22:18:32,211 INFO orderedgrouped.MergeManager (MergeManager.java:getInitialMemoryRequirement(310)) - Initial Shuffle Memory Required: 634178752, based on INPUT_BUFFER_factor: 0.9
2016-05-17 22:18:32,211 INFO orderedgrouped.MergeManager (MergeManager.java:getInitialMemoryRequirement(320)) - Initial Memory required for final merged output: 0, using factor: 0.0
2016-05-17 22:18:32,211 INFO resources.MemoryDistributor$RequestorInfo (MemoryDistributor.java:<init>(233)) - Received request: 634178752, type: INPUT, componentVertexName: B36EAAFFEAE6410589B338EC7841B27C
2016-05-17 22:18:32,212 INFO impl.TaskImpl (TaskImpl.java:getTaskAttemptTezEvents(481)) - TaskAttempt:attempt_1463523511396_0001_1_02_000000_0 sent events: (0-2)
2016-05-17 22:18:32,212 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(438)) - Initializing Output using OutputSpec: { destinationVertexName=42695EBD27C342B7ACC27BC70682EE53, physicalEdgeCount=0, outputClassName=org.apache.tez.mapreduce.output.MROutput }
2016-05-17 22:18:32,211 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initializeLogicalIOProcessor(488)) - Initializing processor, processorClassName=cascading.flow.tez.FlowProcessor
2016-05-17 22:18:32,213 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(403)) - Initialized Input with src edge: B36EAAFFEAE6410589B338EC7841B27C
2016-05-17 22:18:32,214 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:createOutput(565)) - Creating Output
2016-05-17 22:18:32,214 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(446)) - Initializing Output with dest edge: 42695EBD27C342B7ACC27BC70682EE53
2016-05-17 22:18:32,215 INFO output.MROutput (MROutput.java:initialize(347)) - Initializing Simple Output
2016-05-17 22:18:32,215 INFO resources.MemoryDistributor$RequestorInfo (MemoryDistributor.java:<init>(233)) - Received request: 0, type: OUTPUT, componentVertexName: 42695EBD27C342B7ACC27BC70682EE53
2016-05-17 22:18:32,216 INFO tez.FlowProcessor (FlowProcessor.java:initialize(88)) - cascading version: 3.1.0
2016-05-17 22:18:32,226 INFO tez.FlowProcessor (FlowProcessor.java:initialize(94)) - flow node id: E15CD4CB404340CA9C0C4E21BE696455, ordinal: 2
2016-05-17 22:18:32,226 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: E15CD4CB404340CA9C0C4E21BE696455, mem on start (mb), free: 301, total: 528, max: 672
2016-05-17 22:18:32,226 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initializeLogicalIOProcessor(491)) - Initialized processor, processorClassName=cascading.flow.tez.FlowProcessor
2016-05-17 22:18:32,227 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(220)) - Waiting for 3 initializers to finish
2016-05-17 22:18:32,227 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(220)) - Waiting for 2 initializers to finish
2016-05-17 22:18:32,227 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(220)) - Waiting for 1 initializers to finish
2016-05-17 22:18:32,231 INFO output.MROutput (MROutput.java:initialize(418)) - Initialized Simple Output, using_new_api: false
2016-05-17 22:18:32,231 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(452)) - Initialized Output with dest edge: 42695EBD27C342B7ACC27BC70682EE53
2016-05-17 22:18:32,232 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(233)) - All initializers finished
2016-05-17 22:18:32,232 INFO resources.MemoryDistributor (MemoryDistributor.java:makeInitialAllocations(124)) - Using Allocator class: org.apache.tez.runtime.library.resources.WeightedScalingMemoryDistributor
2016-05-17 22:18:32,232 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:initialProcessMemoryRequestContext(171)) - ScaleFactor: 12, for type: SORTED_MERGED_INPUT
2016-05-17 22:18:32,233 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:initialProcessMemoryRequestContext(171)) - ScaleFactor: 12, for type: SORTED_MERGED_INPUT
2016-05-17 22:18:32,233 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:getRequestTypeForClass(197)) - Falling back to RequestType.OTHER for class: org.apache.tez.mapreduce.output.MROutput
2016-05-17 22:18:32,233 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:initialProcessMemoryRequestContext(171)) - ScaleFactor: 1, for type: OTHER
2016-05-17 22:18:32,234 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:computeReservedFraction(262)) - InitialReservationFraction=0.3, AdditionalReservationFractionForIOs=0.045, finalReserveFractionUsed=0.345
2016-05-17 22:18:32,234 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:assignMemory(126)) - Scaling Requests. NumRequests: 3, numScaledRequests: 25, TotalRequested: 1268357504, TotalRequestedScaled: 6.0881160192E8, TotalJVMHeap: 704643072, TotalAvailable: 461541212, TotalRequested/TotalJVMHeap:1.80
2016-05-17 22:18:32,234 INFO resources.MemoryDistributor (MemoryDistributor.java:makeInitialAllocations(138)) - Informing: INPUT, D62C16529B4C47909B52CAE72AF84E87, org.apache.tez.runtime.library.input.OrderedGroupedKVInput: requested=634178752, allocated=230770606
2016-05-17 22:18:32,234 INFO resources.MemoryDistributor (MemoryDistributor.java:makeInitialAllocations(138)) - Informing: INPUT, B36EAAFFEAE6410589B338EC7841B27C, org.apache.tez.runtime.library.input.OrderedGroupedKVInput: requested=634178752, allocated=230770606
2016-05-17 22:18:32,234 INFO resources.MemoryDistributor (MemoryDistributor.java:makeInitialAllocations(138)) - Informing: OUTPUT, 42695EBD27C342B7ACC27BC70682EE53, org.apache.tez.mapreduce.output.MROutput: requested=0, allocated=0
2016-05-17 22:18:32,235 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(255)) - Starting Inputs/Outputs
2016-05-17 22:18:32,235 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(268)) - Input: D62C16529B4C47909B52CAE72AF84E87 being auto started by the framework. Subsequent instances will not be auto-started
2016-05-17 22:18:32,235 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(419)) - Starting Input with src edge: D62C16529B4C47909B52CAE72AF84E87
2016-05-17 22:18:32,237 INFO common.TezRuntimeUtils (TezRuntimeUtils.java:instantiateCombiner(65)) - No combiner specified via tez.runtime.combiner.class. Combiner will not be used
2016-05-17 22:18:32,237 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(268)) - Input: B36EAAFFEAE6410589B338EC7841B27C being auto started by the framework. Subsequent instances will not be auto-started
2016-05-17 22:18:32,237 INFO orderedgrouped.Shuffle (Shuffle.java:<init>(182)) - Shuffle assigned with 1 inputs, codec: NoneifileReadAhead: true
2016-05-17 22:18:32,238 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:<init>(158)) - ShuffleScheduler running for sourceVertex: D62C16529B4C47909B52CAE72AF84E87 with configuration: maxFetchFailuresBeforeReporting=5, reportReadErrorImmediately=true, maxFailedUniqueFetches=1, abortFailureLimit=30, maxMapRuntime=0, maxTaskOutputAtOnce=20
2016-05-17 22:18:32,238 INFO orderedgrouped.MergeManager (MergeManager.java:<init>(221)) - InitialRequest: ShuffleMem=634178752, postMergeMem=0, RuntimeTotalAvailable=230770606. Updated to: ShuffleMem=230770606, postMergeMem: 0
2016-05-17 22:18:32,238 INFO orderedgrouped.MergeManager (MergeManager.java:<init>(252)) - MergerManager: memoryLimit=230770606, maxSingleShuffleLimit=57692652, mergeThreshold=207693536, ioSortFactor=100, memToMemMergeOutputsThreshold=100
2016-05-17 22:18:32,238 INFO orderedgrouped.Shuffle (Shuffle.java:<init>(233)) - Num fetchers being started: 1
2016-05-17 22:18:32,237 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(289)) - Num IOs determined for AutoStart: 2
2016-05-17 22:18:32,237 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(419)) - Starting Input with src edge: B36EAAFFEAE6410589B338EC7841B27C
2016-05-17 22:18:32,239 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(291)) - Waiting for 2 IOs to start
2016-05-17 22:18:32,242 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(421)) - Started Input with src edge: D62C16529B4C47909B52CAE72AF84E87
2016-05-17 22:18:32,242 INFO common.TezRuntimeUtils (TezRuntimeUtils.java:instantiateCombiner(65)) - No combiner specified via tez.runtime.combiner.class. Combiner will not be used
2016-05-17 22:18:32,243 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(291)) - Waiting for 1 IOs to start
2016-05-17 22:18:32,243 INFO orderedgrouped.Shuffle (Shuffle.java:<init>(182)) - Shuffle assigned with 1 inputs, codec: NoneifileReadAhead: true
2016-05-17 22:18:32,243 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:<init>(158)) - ShuffleScheduler running for sourceVertex: B36EAAFFEAE6410589B338EC7841B27C with configuration: maxFetchFailuresBeforeReporting=5, reportReadErrorImmediately=true, maxFailedUniqueFetches=1, abortFailureLimit=30, maxMapRuntime=0, maxTaskOutputAtOnce=20
2016-05-17 22:18:32,243 INFO orderedgrouped.MergeManager (MergeManager.java:<init>(221)) - InitialRequest: ShuffleMem=634178752, postMergeMem=0, RuntimeTotalAvailable=230770606. Updated to: ShuffleMem=230770606, postMergeMem: 0
2016-05-17 22:18:32,244 INFO orderedgrouped.MergeManager (MergeManager.java:<init>(252)) - MergerManager: memoryLimit=230770606, maxSingleShuffleLimit=57692652, mergeThreshold=207693536, ioSortFactor=100, memToMemMergeOutputsThreshold=100
2016-05-17 22:18:32,245 INFO orderedgrouped.Shuffle (Shuffle.java:<init>(233)) - Num fetchers being started: 1
2016-05-17 22:18:32,249 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(421)) - Started Input with src edge: B36EAAFFEAE6410589B338EC7841B27C
2016-05-17 22:18:32,249 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(304)) - AutoStartComplete
2016-05-17 22:18:32,250 INFO task.TezTaskRunner (TezTaskRunner.java:run(175)) - Running task, taskAttemptId=attempt_1463523511396_0001_1_02_000000_0
2016-05-17 22:18:32,252 INFO orderedgrouped.ShuffleInputEventHandlerOrderedGrouped (ShuffleInputEventHandlerOrderedGrouped.java:processDataMovementEvent(84)) - DME srcIdx: 0, targetIdx: 0, attemptNum: 0, payload: [host: ip-10-123-148-208, port: 0, pathComponent: attempt_1463523511396_0001_1_01_000000_0_10024, runDuration: 440421, ]
2016-05-17 22:18:32,253 INFO orderedgrouped.ShuffleInputEventHandlerOrderedGrouped (ShuffleInputEventHandlerOrderedGrouped.java:processDataMovementEvent(84)) - DME srcIdx: 0, targetIdx: 0, attemptNum: 0, payload: [host: ip-10-123-148-208, port: 0, pathComponent: attempt_1463523511396_0001_1_00_000000_0_10023, runDuration: 447704, ]
2016-05-17 22:18:32,255 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:logIndividualFetchComplete(250)) - Completed fetch for attempt: InputAttemptIdentifier [inputIdentifier=InputIdentifier [inputIndex=0], attemptNumber=0, pathComponent=attempt_1463523511396_0001_1_01_000000_0_10024] to DISK_DIRECT, CompressedSize=95, DecompressedSize=91,EndTime=1463523512255, TimeTaken=1, Rate=0.09 MB/s
2016-05-17 22:18:32,255 INFO graph.Hadoop2TezStreamGraph (Hadoop2TezStreamGraph.java:buildGraph(151)) - using streamed source: CoGroup(lower*upper)[by: lower:[{1}:'num'] upper:[{1}:'num']]
2016-05-17 22:18:32,256 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:copySucceeded(213)) - All inputs fetched for input vertex : D62C16529B4C47909B52CAE72AF84E87
2016-05-17 22:18:32,256 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:logProgress(263)) - copy(1 of 1. Transfer rate (CumulativeDataFetched/TimeSinceInputStarted)) 0.00 MB/s)
2016-05-17 22:18:32,256 INFO tez.FlowProcessor (FlowProcessor.java:run(121)) - sourcing from: CoGroup(lower*upper)[by: lower:[{1}:'num'] upper:[{1}:'num']] streamed: true, id: 89F46DD398BA43169B2B67B05CFB2EBB
2016-05-17 22:18:32,257 INFO tez.FlowProcessor (FlowProcessor.java:run(124)) - sinking to: Hfs["TextLine[['line']->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.5.x-tez-0.6.x/build/test/output/hadoop2-tez/bufferpipesplatform/testJoinerClosure/cogroup"], id: 42695EBD27C342B7ACC27BC70682EE53
2016-05-17 22:18:32,257 INFO element.TezSinkStage (TezSinkStage.java:prepare(57)) - calling MROutput#start() on: Hfs["TextLine[['line']->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.5.x-tez-0.6.x/build/test/output/hadoop2-tez/bufferpipesplatform/testJoinerClosure/cogroup"]
2016-05-17 22:18:32,258 INFO element.TezGroupGate (TezGroupGate.java:prepare(90)) - calling OrderedGroupedKVInput#start() on: CoGroup(lower*upper)[by: lower:[{1}:'num'] upper:[{1}:'num']] 89F46DD398BA43169B2B67B05CFB2EBB, for 2 inputs
2016-05-17 22:18:32,258 INFO element.TezGroupGate (TezGroupGate.java:prepare(90)) - calling OrderedGroupedKVInput#start() on: CoGroup(lower*upper)[by: lower:[{1}:'num'] upper:[{1}:'num']] 89F46DD398BA43169B2B67B05CFB2EBB, for 2 inputs
2016-05-17 22:18:32,257 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:freeHost(529)) - ip-10-123-148-208:0 freed by fetcher [D62C16529B4C47909B52CAE72AF84E87] #4 in 3ms
2016-05-17 22:18:32,258 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:logIndividualFetchComplete(250)) - Completed fetch for attempt: InputAttemptIdentifier [inputIdentifier=InputIdentifier [inputIndex=0], attemptNumber=0, pathComponent=attempt_1463523511396_0001_1_00_000000_0_10023] to DISK_DIRECT, CompressedSize=95, DecompressedSize=91,EndTime=1463523512257, TimeTaken=3, Rate=0.03 MB/s
2016-05-17 22:18:32,259 INFO orderedgrouped.Shuffle (Shuffle.java:cleanupFetchers(394)) - Shutdown..fetcher [D62C16529B4C47909B52CAE72AF84E87] #4, status:false, isInterrupted:false
2016-05-17 22:18:32,259 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:copySucceeded(213)) - All inputs fetched for input vertex : B36EAAFFEAE6410589B338EC7841B27C
2016-05-17 22:18:32,259 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(854)) - finalMerge called with 0 in-memory map-outputs and 1 on-disk map-outputs
2016-05-17 22:18:32,260 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(951)) - Merging 1 files, 95 bytes from disk
2016-05-17 22:18:32,261 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(966)) - Merging 0 segments, 0 bytes from memory into reduce
2016-05-17 22:18:32,261 INFO impl.TezMerger (TezMerger.java:merge(560)) - Merging 1 sorted segments
2016-05-17 22:18:32,261 INFO impl.TezMerger (TezMerger.java:merge(665)) - Down to the last merge-pass, with 1 segments left of total size: 78 bytes
2016-05-17 22:18:32,261 INFO orderedgrouped.Shuffle (Shuffle.java:call(381)) - merge complete for input vertex : D62C16529B4C47909B52CAE72AF84E87
2016-05-17 22:18:32,263 INFO orderedgrouped.Shuffle (Shuffle.java:onSuccess(488)) - Shuffle Runner thread complete
2016-05-17 22:18:32,263 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:logProgress(263)) - copy(1 of 1. Transfer rate (CumulativeDataFetched/TimeSinceInputStarted)) 0.00 MB/s)
2016-05-17 22:18:32,264 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:freeHost(529)) - ip-10-123-148-208:0 freed by fetcher [B36EAAFFEAE6410589B338EC7841B27C] #5 in 10ms
2016-05-17 22:18:32,265 INFO orderedgrouped.Shuffle (Shuffle.java:cleanupFetchers(394)) - Shutdown..fetcher [B36EAAFFEAE6410589B338EC7841B27C] #5, status:false, isInterrupted:false
2016-05-17 22:18:32,266 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(854)) - finalMerge called with 0 in-memory map-outputs and 1 on-disk map-outputs
2016-05-17 22:18:32,267 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(951)) - Merging 1 files, 95 bytes from disk
2016-05-17 22:18:32,267 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(966)) - Merging 0 segments, 0 bytes from memory into reduce
2016-05-17 22:18:32,267 INFO impl.TezMerger (TezMerger.java:merge(560)) - Merging 1 sorted segments
2016-05-17 22:18:32,267 INFO impl.TezMerger (TezMerger.java:merge(665)) - Down to the last merge-pass, with 1 segments left of total size: 78 bytes
2016-05-17 22:18:32,267 INFO orderedgrouped.Shuffle (Shuffle.java:call(381)) - merge complete for input vertex : B36EAAFFEAE6410589B338EC7841B27C
2016-05-17 22:18:32,268 INFO orderedgrouped.Shuffle (Shuffle.java:onSuccess(488)) - Shuffle Runner thread complete
2016-05-17 22:18:32,268 INFO collect.SpillableTupleList (SpillableTupleList.java:getCodecClass(89)) - attempting to load codec: org.apache.hadoop.io.compress.GzipCodec
2016-05-17 22:18:32,268 INFO collect.SpillableTupleList (SpillableTupleList.java:getCodecClass(94)) - found codec: org.apache.hadoop.io.compress.GzipCodec
2016-05-17 22:18:32,272 INFO tez.FlowProcessor (FlowProcessor.java:waitForInputsReady(202)) - flow node id: E15CD4CB404340CA9C0C4E21BE696455, all 2 inputs ready in: 00:00:00.000
2016-05-17 22:18:32,274 INFO common.ValuesIterator (ValuesIterator.java:<init>(83)) - keyDeserializer=cascading.tuple.hadoop.io.KeyTupleDeserializer@1f8dd5bf; valueDeserializer=cascading.tuple.hadoop.io.ValueTupleDeserializer@18cbfec8
2016-05-17 22:18:32,275 INFO common.ValuesIterator (ValuesIterator.java:<init>(83)) - keyDeserializer=cascading.tuple.hadoop.io.KeyTupleDeserializer@e372a9a; valueDeserializer=cascading.tuple.hadoop.io.ValueTupleDeserializer@47cdebc0
2016-05-17 22:18:32,280 INFO output.MROutput (MROutput.java:flush(534)) - Flushing Simple Output
2016-05-17 22:18:32,280 INFO output.MROutput (MROutput.java:flush(544)) - Flushed Simple Output
2016-05-17 22:18:32,281 INFO app.TaskAttemptListenerImpTezDag (TaskAttemptListenerImpTezDag.java:canCommit(353)) - Commit go/no-go request from attempt_1463523511396_0001_1_02_000000_0
2016-05-17 22:18:32,282 INFO impl.TaskImpl (TaskImpl.java:canCommit(750)) - attempt_1463523511396_0001_1_02_000000_0 given a go for committing the task output.
2016-05-17 22:18:32,283 INFO task.TezTaskRunner (TezTaskRunner.java:run(177)) - Closing task, taskAttemptId=attempt_1463523511396_0001_1_02_000000_0
2016-05-17 22:18:32,283 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: E15CD4CB404340CA9C0C4E21BE696455, mem on close (mb), free: 294, total: 528, max: 672
2016-05-17 22:18:32,283 INFO orderedgrouped.Shuffle (Shuffle.java:shutdown(320)) - Shutting down Shuffle for source: D62C16529B4C47909B52CAE72AF84E87
2016-05-17 22:18:32,284 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(854)) - finalMerge called with 0 in-memory map-outputs and 0 on-disk map-outputs
2016-05-17 22:18:32,285 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(951)) - Merging 0 files, 0 bytes from disk
2016-05-17 22:18:32,285 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(966)) - Merging 0 segments, 0 bytes from memory into reduce
2016-05-17 22:18:32,285 INFO impl.TezMerger (TezMerger.java:merge(560)) - Merging 0 sorted segments
2016-05-17 22:18:32,286 INFO impl.TezMerger (TezMerger.java:merge(562)) - Nothing to merge. Returning an empty iterator
2016-05-17 22:18:32,286 INFO orderedgrouped.Shuffle (Shuffle.java:shutdown(320)) - Shutting down Shuffle for source: B36EAAFFEAE6410589B338EC7841B27C
2016-05-17 22:18:32,289 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(854)) - finalMerge called with 0 in-memory map-outputs and 0 on-disk map-outputs
2016-05-17 22:18:32,290 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(951)) - Merging 0 files, 0 bytes from disk
2016-05-17 22:18:32,291 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(966)) - Merging 0 segments, 0 bytes from memory into reduce
2016-05-17 22:18:32,291 INFO impl.TezMerger (TezMerger.java:merge(560)) - Merging 0 sorted segments
2016-05-17 22:18:32,291 INFO impl.TezMerger (TezMerger.java:merge(562)) - Nothing to merge. Returning an empty iterator
2016-05-17 22:18:32 Completed running task attempt: attempt_1463523511396_0001_1_02_000000_0
2016-05-17 22:18:32,293 INFO task.TezTaskRunner (TezTaskRunner.java:run(181)) - Task completed, taskAttemptId=attempt_1463523511396_0001_1_02_000000_0, fatalErrorOccurred=false
2016-05-17 22:18:32,294 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:cleanup(712)) - Final Counters : Counters: 46 [[File System Counters FILE_BYTES_READ=254, FILE_BYTES_WRITTEN=52, FILE_READ_OPS=0, FILE_LARGE_READ_OPS=0, FILE_WRITE_OPS=0][org.apache.tez.common.counters.TaskCounter REDUCE_INPUT_GROUPS=8, REDUCE_INPUT_RECORDS=10, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=10, NUM_SHUFFLED_INPUTS=2, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=2, GC_TIME_MILLIS=0, CPU_MILLISECONDS=0, PHYSICAL_MEMORY_BYTES=0, VIRTUAL_MEMORY_BYTES=0, COMMITTED_HEAP_BYTES=553648128, OUTPUT_RECORDS=5, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=190, SHUFFLE_BYTES=190, SHUFFLE_BYTES_DECOMPRESSED=182, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=190, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=22, MERGE_PHASE_TIME=24, FIRST_EVENT_RECEIVED=16, LAST_EVENT_RECEIVED=11][Shuffle Errors BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0][cascading.flow.SliceCounters Process_Begin_Time=1463523512272, Process_Duration=11, Process_End_Time=1463523512283, Read_Duration=1, Tuples_Read=10, Tuples_Written=5, Write_Duration=0][cascading.flow.StepCounters Tuples_Written=5]]
2016-05-17 22:18:32,295 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(164)) - Redirecting log file based on addend:
2016-05-17 22:18:32,295 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(179)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-17 22:18:32,295 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1463523511396_0001_1][Event:TASK_ATTEMPT_FINISHED]: vertexName=E15CD4CB404340CA9C0C4E21BE696455, taskAttemptId=attempt_1463523511396_0001_1_02_000000_0, startTime=1463523512198, finishTime=1463523512294, timeTaken=96, status=SUCCEEDED, errorEnum=, diagnostics=, counters=Counters: 46, File System Counters, FILE_BYTES_READ=254, FILE_BYTES_WRITTEN=52, FILE_READ_OPS=0, FILE_LARGE_READ_OPS=0, FILE_WRITE_OPS=0, org.apache.tez.common.counters.TaskCounter, REDUCE_INPUT_GROUPS=8, REDUCE_INPUT_RECORDS=10, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=10, NUM_SHUFFLED_INPUTS=2, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=2, GC_TIME_MILLIS=0, CPU_MILLISECONDS=0, PHYSICAL_MEMORY_BYTES=0, VIRTUAL_MEMORY_BYTES=0, COMMITTED_HEAP_BYTES=553648128, OUTPUT_RECORDS=5, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=190, SHUFFLE_BYTES=190, SHUFFLE_BYTES_DECOMPRESSED=182, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=190, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=22, MERGE_PHASE_TIME=24, FIRST_EVENT_RECEIVED=16, LAST_EVENT_RECEIVED=11, Shuffle Errors, BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0, cascading.flow.SliceCounters, Process_Begin_Time=1463523512272, Process_Duration=11, Process_End_Time=1463523512283, Read_Duration=1, Tuples_Read=10, Tuples_Written=5, Write_Duration=0, cascading.flow.StepCounters, Tuples_Written=5
2016-05-17 22:18:32,295 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:handle(692)) - attempt_1463523511396_0001_1_02_000000_0 TaskAttempt Transitioned from RUNNING to SUCCEEDED due to event TA_DONE
2016-05-17 22:18:32,297 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event TASK_ATTEMPT_FINISHED to history file
2016-05-17 22:18:32,297 INFO impl.TaskImpl (TaskImpl.java:transition(1052)) - Task succeeded with attempt attempt_1463523511396_0001_1_02_000000_0
2016-05-17 22:18:32,298 INFO rm.TaskSchedulerEventHandler (TaskSchedulerEventHandler.java:handleEvent(149)) - Processing the event EventType: S_TA_ENDED
2016-05-17 22:18:32,298 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event TASK_FINISHED to history file
2016-05-17 22:18:32,298 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1463523511396_0001_1][Event:TASK_FINISHED]: vertexName=E15CD4CB404340CA9C0C4E21BE696455, taskId=task_1463523511396_0001_1_02_000000, startTime=1463523512198, finishTime=1463523512297, timeTaken=99, status=SUCCEEDED, successfulAttemptID=attempt_1463523511396_0001_1_02_000000_0, diagnostics=, counters=Counters: 46, File System Counters, FILE_BYTES_READ=254, FILE_BYTES_WRITTEN=52, FILE_READ_OPS=0, FILE_LARGE_READ_OPS=0, FILE_WRITE_OPS=0, org.apache.tez.common.counters.TaskCounter, REDUCE_INPUT_GROUPS=8, REDUCE_INPUT_RECORDS=10, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=10, NUM_SHUFFLED_INPUTS=2, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=2, GC_TIME_MILLIS=0, CPU_MILLISECONDS=0, PHYSICAL_MEMORY_BYTES=0, VIRTUAL_MEMORY_BYTES=0, COMMITTED_HEAP_BYTES=553648128, OUTPUT_RECORDS=5, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=190, SHUFFLE_BYTES=190, SHUFFLE_BYTES_DECOMPRESSED=182, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=190, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=22, MERGE_PHASE_TIME=24, FIRST_EVENT_RECEIVED=16, LAST_EVENT_RECEIVED=11, Shuffle Errors, BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0, cascading.flow.SliceCounters, Process_Begin_Time=1463523512272, Process_Duration=11, Process_End_Time=1463523512283, Read_Duration=1, Tuples_Read=10, Tuples_Written=5, Write_Duration=0, cascading.flow.StepCounters, Tuples_Written=5
2016-05-17 22:18:32,298 INFO impl.TaskImpl (TaskImpl.java:handle(862)) - task_1463523511396_0001_1_02_000000 Task Transitioned from RUNNING to SUCCEEDED due to event T_ATTEMPT_SUCCEEDED
2016-05-17 22:18:32,298 INFO impl.VertexImpl (VertexImpl.java:transition(3536)) - Num completed Tasks for vertex_1463523511396_0001_1_02 [E15CD4CB404340CA9C0C4E21BE696455] : 1
2016-05-17 22:18:32,299 INFO impl.VertexImpl (VertexImpl.java:checkVertexForCompletion(1674)) - Vertex succeeded: vertex_1463523511396_0001_1_02 [E15CD4CB404340CA9C0C4E21BE696455]
2016-05-17 22:18:32,299 INFO task.ContainerReporter (ContainerReporter.java:call(55)) - Attempting to fetch new task for container container_1463523511396_0001_00_000003
2016-05-17 22:18:32,300 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event VERTEX_FINISHED to history file
2016-05-17 22:18:32,300 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1463523511396_0001_1][Event:VERTEX_FINISHED]: vertexName=E15CD4CB404340CA9C0C4E21BE696455, vertexId=vertex_1463523511396_0001_1_02, initRequestedTime=1463523511628, initedTime=1463523511640, startRequestedTime=1463523511668, startedTime=1463523511668, finishTime=1463523512299, timeTaken=631, status=SUCCEEDED, diagnostics=, counters=Counters: 46, File System Counters, FILE_BYTES_READ=254, FILE_BYTES_WRITTEN=52, FILE_READ_OPS=0, FILE_LARGE_READ_OPS=0, FILE_WRITE_OPS=0, org.apache.tez.common.counters.TaskCounter, REDUCE_INPUT_GROUPS=8, REDUCE_INPUT_RECORDS=10, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=10, NUM_SHUFFLED_INPUTS=2, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=2, GC_TIME_MILLIS=0, CPU_MILLISECONDS=0, PHYSICAL_MEMORY_BYTES=0, VIRTUAL_MEMORY_BYTES=0, COMMITTED_HEAP_BYTES=553648128, OUTPUT_RECORDS=5, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=190, SHUFFLE_BYTES=190, SHUFFLE_BYTES_DECOMPRESSED=182, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=190, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=22, MERGE_PHASE_TIME=24, FIRST_EVENT_RECEIVED=16, LAST_EVENT_RECEIVED=11, Shuffle Errors, BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0, cascading.flow.SliceCounters, Process_Begin_Time=1463523512272, Process_Duration=11, Process_End_Time=1463523512283, Read_Duration=1, Tuples_Read=10, Tuples_Written=5, Write_Duration=0, cascading.flow.StepCounters, Tuples_Written=5, vertexStats=firstTaskStartTime=1463523512198, firstTasksToStart=[ task_1463523511396_0001_1_02_000000 ], lastTaskFinishTime=1463523512294, lastTasksToFinish=[ task_1463523511396_0001_1_02_000000 ], minTaskDuration=96, maxTaskDuration=96, avgTaskDuration=96.0, numSuccessfulTasks=1, shortestDurationTasks=[ task_1463523511396_0001_1_02_000000 ], longestDurationTasks=[ task_1463523511396_0001_1_02_000000 ], vertexTaskStats={numFailedTasks=0, numSucceededTasks=1, numKilledTaskAttempts=0, numKilledTasks=0, numFailedTaskAttempts=0, numCompletedTasks=1}
2016-05-17 22:18:32,300 INFO impl.VertexImpl (VertexImpl.java:handle(1564)) - vertex_1463523511396_0001_1_02 [E15CD4CB404340CA9C0C4E21BE696455] transitioned from RUNNING to SUCCEEDED due to event V_TASK_COMPLETED
2016-05-17 22:18:32,300 INFO container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1463523511396_0001_00_000003 transitioned from RUNNING to IDLE via event C_TA_SUCCEEDED
2016-05-17 22:18:32,301 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1463523511396_0001_1][Event:CONTAINER_STOPPED]: containerId=container_1463523511396_0001_00_000003, stoppedTime=1463523512301, exitStatus=0
2016-05-17 22:18:32,301 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event CONTAINER_STOPPED to history file
2016-05-17 22:18:32,301 INFO container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1463523511396_0001_00_000003 transitioned from IDLE to STOP_REQUESTED via event C_STOP_REQUEST
2016-05-17 22:18:32,301 INFO impl.DAGImpl (DAGImpl.java:transition(1747)) - Vertex vertex_1463523511396_0001_1_02 [E15CD4CB404340CA9C0C4E21BE696455] completed., numCompletedVertices=3, numSuccessfulVertices=3, numFailedVertices=0, numKilledVertices=0, numVertices=3
2016-05-17 22:18:32,302 INFO impl.DAGImpl (DAGImpl.java:commitOrAbortOutputs(831)) - Calling DAG commit/abort for dag: dag_1463523511396_0001_1
2016-05-17 22:18:32,302 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1463523511396_0001_1][Event:DAG_COMMIT_STARTED]: dagID=dag_1463523511396_0001_1
2016-05-17 22:18:32,302 INFO impl.DAGImpl (DAGImpl.java:commitOrAbortOutputs(871)) - No output committers for vertex: vertex_1463523511396_0001_1_00 [B36EAAFFEAE6410589B338EC7841B27C]
2016-05-17 22:18:32,302 INFO impl.DAGImpl (DAGImpl.java:commitOrAbortOutputs(871)) - No output committers for vertex: vertex_1463523511396_0001_1_01 [D62C16529B4C47909B52CAE72AF84E87]
2016-05-17 22:18:32,302 INFO impl.DAGImpl (DAGImpl.java:commitOrAbortOutputs(892)) - Committing output: 42695EBD27C342B7ACC27BC70682EE53 for vertex: vertex_1463523511396_0001_1_02 [E15CD4CB404340CA9C0C4E21BE696455]
2016-05-17 22:18:32,309 INFO app.TaskAttemptListenerImpTezDag (TaskAttemptListenerImpTezDag.java:getTask(213)) - Container with id: container_1463523511396_0001_00_000003 is valid, but no longer registered, and will be killed
2016-05-17 22:18:32,309 INFO task.ContainerReporter (ContainerReporter.java:call(65)) - Got TaskUpdate for containerId= container_1463523511396_0001_00_000003: 10 ms after starting to poll. TaskInfo: shouldDie: true
2016-05-17 22:18:32,310 INFO task.TezChild (TezChild.java:run(207)) - ContainerTask returned shouldDie=true for container container_1463523511396_0001_00_000003, Exiting
2016-05-17 22:18:32,310 INFO task.TezChild (TezChild.java:shutdown(343)) - Shutdown invoked for container container_1463523511396_0001_00_000003
2016-05-17 22:18:32,310 INFO task.TezChild (TezChild.java:shutdown(345)) - Shutting down container container_1463523511396_0001_00_000003
2016-05-17 22:18:32,311 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(270)) - ContainerExecutionResult for: container_1463523511396_0001_00_000003 = ContainerExecutionResult{exitStatus=SUCCESS, throwable=null, errorMessage='Asked to die by the AM'}
2016-05-17 22:18:32,311 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(274)) - Container: container_1463523511396_0001_00_000003 completed successfully
2016-05-17 22:18:32,315 INFO recovery.RecoveryService (RecoveryService.java:handle(306)) - DAG completed, dagId=dag_1463523511396_0001_1, queueSize=0
2016-05-17 22:18:32,315 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event DAG_FINISHED to history file
2016-05-17 22:18:32,316 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1463523511396_0001_1][Event:DAG_FINISHED]: dagId=dag_1463523511396_0001_1, startTime=1463523511619, finishTime=1463523512311, timeTaken=692, status=SUCCEEDED, diagnostics=, counters=Counters: 55, org.apache.tez.common.counters.DAGCounter, NUM_SUCCEEDED_TASKS=3, TOTAL_LAUNCHED_TASKS=3, RACK_LOCAL_TASKS=2, File System Counters, FILE_BYTES_READ=502, FILE_BYTES_WRITTEN=2640, FILE_READ_OPS=0, FILE_LARGE_READ_OPS=0, FILE_WRITE_OPS=0, org.apache.tez.common.counters.TaskCounter, REDUCE_INPUT_GROUPS=8, REDUCE_INPUT_RECORDS=10, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=20, NUM_SHUFFLED_INPUTS=2, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=2, GC_TIME_MILLIS=732, CPU_MILLISECONDS=0, PHYSICAL_MEMORY_BYTES=0, VIRTUAL_MEMORY_BYTES=0, COMMITTED_HEAP_BYTES=1660944384, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=15, OUTPUT_BYTES=150, OUTPUT_BYTES_WITH_OVERHEAD=182, OUTPUT_BYTES_PHYSICAL=190, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=190, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_BYTES=190, SHUFFLE_BYTES_DECOMPRESSED=182, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=190, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=22, MERGE_PHASE_TIME=24, FIRST_EVENT_RECEIVED=16, LAST_EVENT_RECEIVED=11, Shuffle Errors, BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0, cascading.flow.SliceCounters, Process_Begin_Time=4390570536545, Process_Duration=31, Process_End_Time=4390570536576, Read_Duration=19, Tuples_Read=20, Tuples_Written=15, Write_Duration=2, cascading.flow.StepCounters, Tuples_Read=10, Tuples_Written=5
2016-05-17 22:18:32,316 INFO impl.DAGImpl (DAGImpl.java:finished(1180)) - DAG: dag_1463523511396_0001_1 finished with state: SUCCEEDED
2016-05-17 22:18:32,317 INFO impl.DAGImpl (DAGImpl.java:handle(973)) - dag_1463523511396_0001_1 transitioned from RUNNING to SUCCEEDED due to event DAG_VERTEX_COMPLETED
2016-05-17 22:18:32,317 INFO container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1463523511396_0001_00_000003 transitioned from STOP_REQUESTED to COMPLETED via event C_COMPLETED
2016-05-17 22:18:32,317 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:stop(242)) - Ignoring stop request for containerId: container_1463523511396_0001_00_000003
2016-05-17 22:18:32 Completed Dag: dag_1463523511396_0001_1
2016-05-17 22:18:32,317 INFO app.DAGAppMaster (DAGAppMaster.java:handle(579)) - DAG completed, dagId=dag_1463523511396_0001_1, dagState=SUCCEEDED
2016-05-17 22:18:32,317 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(164)) - Redirecting log file based on addend: dag_1463523511396_0001_1_post
2016-05-17 22:18:32,318 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(179)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-17 22:18:32,318 INFO app.DAGAppMaster (DAGAppMaster.java:handle(625)) - Waiting for next DAG to be submitted.
2016-05-17 22:18:32,326 INFO client.TezClient (TezClient.java:stop(495)) - Shutting down Tez Session, sessionName=(1/1) ...testJoinerClosure/cogroup, applicationId=application_1463523511396_0001
2016-05-17 22:18:32,329 INFO client.DAGClientHandler (DAGClientHandler.java:shutdownAM(117)) - Received message to shutdown AM
2016-05-17 22:18:32,329 INFO rm.TaskSchedulerEventHandler (TaskSchedulerEventHandler.java:setShouldUnregisterFlag(580)) - TaskScheduler notified that it should unregister from RM
2016-05-17 22:18:32,329 INFO app.DAGAppMaster (DAGAppMaster.java:shutdownTezAM(1088)) - No current running DAG, shutting down the AM
2016-05-17 22:18:32,329 INFO app.DAGAppMaster (DAGAppMaster.java:shutdown(692)) - DAGAppMasterShutdownHandler invoked
2016-05-17 22:18:32,330 INFO app.DAGAppMaster (DAGAppMaster.java:shutdown(701)) - Handling DAGAppMaster shutdown
2016-05-17 22:18:32,331 INFO app.DAGAppMaster (DAGAppMaster.java:run(721)) - Sleeping for 5 seconds before shutting down
2016-05-17 22:18:32,332 INFO history.HistoryEventHandler (HistoryEventHandler.java:serviceStop(79)) - Stopping HistoryEventHandler
2016-05-17 22:18:32,332 INFO recovery.RecoveryService (RecoveryService.java:serviceStop(189)) - Stopping RecoveryService
2016-05-17 22:18:32,332 INFO recovery.RecoveryService (RecoveryService.java:serviceStop(192)) - Handle the remaining events in queue, queue size=0
2016-05-17 22:18:32,332 INFO recovery.RecoveryService (RecoveryService.java:run(166)) - EventQueue take interrupted. Returning
2016-05-17 22:18:32,332 INFO recovery.RecoveryService (RecoveryService.java:serviceStop(214)) - Closing Summary Stream
2016-05-17 22:18:32,333 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:serviceStop(118)) - Stopping SimpleHistoryLoggingService, eventQueueBacklog=0
2016-05-17 22:18:32,333 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:run(105)) - EventQueue take interrupted. Returning
2016-05-17 22:18:32,339 INFO app.DAGAppMaster (DAGAppMaster.java:run(1773)) - Completed deletion of tez scratch data dir, path=file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.5.x-tez-0.6.x/build/test/tmp/.staging/66053CB505334C8DB685D7EB09C005D3/D8F904EC7C6D47DFBA81E1EFBF1F555A/.tez/application_1463523511396_0001
2016-05-17 22:18:32,340 INFO util.Hadoop18TapUtil (Hadoop18TapUtil.java:cleanTempPath(226)) - deleting temp path /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.5.x-tez-0.6.x/build/test/output/hadoop2-tez/bufferpipesplatform/testJoinerClosure/cogroup/_temporary
2016-05-17 22:18:32,341 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [lower*upper] completed in: 00:00.971
2016-05-17 22:18:32,426 INFO cascading.PlatformTestCase (PlatformTestCase.java:tearDown(189)) - copying to local /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.5.x-tez-0.6.x/build/test/output//hadoop2-tez/bufferpipesplatform/testJoinerClosure/cogroup
2016-05-17 22:18:32,452 INFO util.Util (Util.java:findMainClass(1374)) - using default application jar, may cause class not found exceptions on the cluster
2016-05-17 22:18:32,452 INFO planner.Hadoop2TezPlanner (Hadoop2TezPlanner.java:initialize(151)) - using application jar: /opt/concurrent/home/teamcity/.gradle/caches/modules-2/files-2.1/cascading/cascading-hadoop2-tez/3.1.0/226af02e24d5195c60e4ebe911d0c7c9780f59f0/cascading-hadoop2-tez-3.1.0.jar
2016-05-17 22:18:32,502 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [test] executed rule registry: NoHashJoinHadoop2TezRuleRegistry, completed as: SUCCESS, in: 00:00.022
2016-05-17 22:18:32,521 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [test] executed rule registry: HashJoinHadoop2TezRuleRegistry, completed as: SUCCESS, in: 00:00.041
2016-05-17 22:18:32,522 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [test] rule registry: NoHashJoinHadoop2TezRuleRegistry, supports assembly with steps: 1, nodes: 2
2016-05-17 22:18:32,522 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [test] rule registry: HashJoinHadoop2TezRuleRegistry, supports assembly with steps: 1, nodes: 2
2016-05-17 22:18:32,523 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [test] rule registry: NoHashJoinHadoop2TezRuleRegistry, result was selected using: 'default comparator: selects plan with fewest steps and fewest nodes'
2016-05-17 22:18:32,540 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [test] tap forcing step to tez local mode: /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.5.x-tez-0.6.x/build/classes/test/data/lhs.txt
2016-05-17 22:18:32,549 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [test] tap forcing step to tez local mode: /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.5.x-tez-0.6.x/build/test/output/hadoop2-tez/bufferpipesplatform/testSimpleBuffer2/simple2
2016-05-17 22:18:32,603 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [test] starting
2016-05-17 22:18:32,603 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [test] source: Hfs["TextLine[['offset', 'line' | long, String]->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.5.x-tez-0.6.x/build/classes/test/data/lhs.txt"]
2016-05-17 22:18:32,604 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [test] sink: Hfs["TextLine[['line']->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.5.x-tez-0.6.x/build/test/output/hadoop2-tez/bufferpipesplatform/testSimpleBuffer2/simple2"]
2016-05-17 22:18:32,604 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [test] parallel execution of steps is enabled: false
2016-05-17 22:18:32,604 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [test] executing total steps: 1
2016-05-17 22:18:32,604 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [test] allocating management threads: 1
2016-05-17 22:18:32,605 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [test] starting step: (1/1) ...testSimpleBuffer2/simple2
2016-05-17 22:18:32,605 WARN flow.Flow (BaseFlow.java:logWarn(1516)) - [test] 'yarn.timeline-service.enabled' is disabled, please enable to capture detailed metrics of completed flows, this may require starting the YARN timeline server daemon
2016-05-17 22:18:32,617 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [test] tez session mode enabled: true
2016-05-17 22:18:32,622 INFO client.TezClient (TezClient.java:<init>(153)) - Tez Client Version: [ component=tez-api, version=0.6.2, revision=c04283bf91fd96823eec37056d5c8b1a92728265, SCM-URL=scm:git:https://git-wip-us.apache.org/repos/asf/tez.git, buildTime=20150804-1436 ]
2016-05-17 22:18:32,623 INFO client.TezClient (TezClient.java:start(333)) - Session mode. Starting session.
2016-05-17 22:18:32,623 INFO client.TezClientUtils (TezClientUtils.java:setupTezJarsLocalResources(159)) - Ignoring 'tez.lib.uris' since 'tez.ignore.lib.uris' is set to true
2016-05-17 22:18:32,632 INFO client.TezClient (TezCommonUtils.java:createTezSystemStagingPath(118)) - Tez system stage directory file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.5.x-tez-0.6.x/build/test/tmp/.staging/5F24C8AC118845E5B6CBE3B6796D5655/DBF10C5DC60F4509983187E52CD42569/.tez/application_1463523512622_0001 doesn't exist and is created
2016-05-17 22:18:32,649 INFO client.LocalClient (LocalClient.java:createDAGAppMaster(328)) - DAGAppMaster thread has been created
2016-05-17 22:18:32,649 INFO client.LocalClient (LocalClient.java:startDAGAppMaster(251)) - DAGAppMaster is not created wait for 100ms...
2016-05-17 22:18:32,649 INFO client.LocalClient (LocalClient.java:run(275)) - Using staging directory: /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.5.x-tez-0.6.x/build/test/tmp/.staging/5F24C8AC118845E5B6CBE3B6796D5655/DBF10C5DC60F4509983187E52CD42569/.tez/application_1463523512622_0001
2016-05-17 22:18:32,653 INFO app.DAGAppMaster (DAGAppMaster.java:<init>(309)) - Created DAGAppMaster for application appattempt_1463523512622_0001_000000, versionInfo=[ component=tez-dag, version=0.6.2, revision=c04283bf91fd96823eec37056d5c8b1a92728265, SCM-URL=scm:git:https://git-wip-us.apache.org/repos/asf/tez.git, buildTime=20150804-1436 ]
2016-05-17 22:18:32,663 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(333)) - Comparing client version with AM version, clientVersion=0.6.2, AMVersion=0.6.2
2016-05-17 22:18:32,664 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(386)) - Adding session token to jobTokenSecretManager for application
2016-05-17 22:18:32,666 INFO node.AMNodeTracker (AMNodeTracker.java:serviceInit(77)) - blacklistDisablePercent is 33, blacklistingEnabled: false, maxTaskFailuresPerNode: 1
2016-05-17 22:18:32,666 INFO history.HistoryEventHandler (HistoryEventHandler.java:serviceInit(50)) - Initializing HistoryEventHandler
2016-05-17 22:18:32,666 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:serviceInit(66)) - Log file location for SimpleHistoryLoggingService not specified, defaulting to containerLogDir=/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.5.x-tez-0.6.x/build/test/tmp/.staging/5F24C8AC118845E5B6CBE3B6796D5655/DBF10C5DC60F4509983187E52CD42569/.tez/application_1463523512622_0001/localmode-log-dir
2016-05-17 22:18:32,666 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:serviceInit(88)) - Initializing SimpleHistoryLoggingService, logFileLocation=/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.5.x-tez-0.6.x/build/test/tmp/.staging/5F24C8AC118845E5B6CBE3B6796D5655/DBF10C5DC60F4509983187E52CD42569/.tez/application_1463523512622_0001/localmode-log-dir/history.txt.appattempt_1463523512622_0001_000000, maxErrors=10
2016-05-17 22:18:32,666 INFO recovery.RecoveryService (RecoveryService.java:serviceInit(109)) - Initializing RecoveryService
2016-05-17 22:18:32,667 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:N/A][Event:AM_LAUNCHED]: appAttemptId=appattempt_1463523512622_0001_000000, appSubmitTime=1463523512652, launchTime=1463523512653
2016-05-17 22:18:32,669 INFO history.HistoryEventHandler (HistoryEventHandler.java:serviceStart(73)) - Starting HistoryEventHandler
2016-05-17 22:18:32,674 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:serviceStart(95)) - Starting SimpleHistoryLoggingService
2016-05-17 22:18:32,677 INFO recovery.RecoveryService (RecoveryService.java:serviceStart(127)) - Starting RecoveryService
2016-05-17 22:18:32,677 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event AM_LAUNCHED to history file
2016-05-17 22:18:32,679 INFO client.DAGClientServer (DAGClientServer.java:serviceStart(91)) - Instantiated DAGClientRPCServer at ip-10-123-148-208.ec2.internal/10.123.148.208:36188
2016-05-17 22:18:32,681 WARN impl.MetricsConfig (MetricsConfig.java:loadFirst(124)) - Cannot locate configuration: tried hadoop-metrics2-dagappmaster.properties,hadoop-metrics2.properties
2016-05-17 22:18:32,683 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:N/A][Event:AM_STARTED]: appAttemptId=appattempt_1463523512622_0001_000000, startTime=1463523512683
2016-05-17 22:18:32,683 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event AM_STARTED to history file
2016-05-17 22:18:32,683 INFO app.DAGAppMaster (DAGAppMaster.java:serviceStart(1644)) - In Session mode. Waiting for DAG over RPC
2016-05-17 22:18:32,749 INFO client.LocalClient (LocalClient.java:startDAGAppMaster(236)) - DAGAppMaster state: IDLE
2016-05-17 22:18:32,749 INFO client.TezClient (TezClient.java:start(367)) - The url to track the Tez Session: N/A
2016-05-17 22:18:32,749 INFO client.TezClient (TezClient.java:submitDAGSession(405)) - Submitting dag to TezSession, sessionName=(1/1) ...testSimpleBuffer2/simple2, applicationId=application_1463523512622_0001, dagName=[5F24C8AC118845E5B6CBE3B6796D5655/DBF10C5DC60F4509983187E52CD42569] test/(1/1) ...testSimpleBuffer2/simple2
2016-05-17 22:18:32,767 INFO app.DAGAppMaster (DAGAppMaster.java:submitDAGToAppMaster(1126)) - Starting DAG submitted via RPC: [5F24C8AC118845E5B6CBE3B6796D5655/DBF10C5DC60F4509983187E52CD42569] test/(1/1) ...testSimpleBuffer2/simple2
2016-05-17 22:18:32,769 INFO app.DAGAppMaster (DAGAppMaster.java:generateDAGVizFile(870)) - Generating DAG graphviz file, dagId=dag_1463523512622_0001_1, filePath=/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.5.x-tez-0.6.x/build/test/tmp/.staging/5F24C8AC118845E5B6CBE3B6796D5655/DBF10C5DC60F4509983187E52CD42569/.tez/application_1463523512622_0001/localmode-log-dir/dag_1463523512622_0001_1.dot
2016-05-17 22:18:32,770 INFO app.DAGAppMaster (DAGAppMaster.java:writePBTextFile(887)) - Writing DAG plan to: /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.5.x-tez-0.6.x/build/test/tmp/.staging/5F24C8AC118845E5B6CBE3B6796D5655/DBF10C5DC60F4509983187E52CD42569/.tez/application_1463523512622_0001/localmode-log-dir/dag_1463523512622_0001_1-tez-dag.pb.txt
2016-05-17 22:18:32,802 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(164)) - Redirecting log file based on addend: dag_1463523512622_0001_1
2016-05-17 22:18:32,803 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(179)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-17 22:18:32,803 INFO app.DAGAppMaster (DAGAppMaster.java:startDAG(2051)) - Running DAG: [5F24C8AC118845E5B6CBE3B6796D5655/DBF10C5DC60F4509983187E52CD42569] test/(1/1) ...testSimpleBuffer2/simple2
2016-05-17 22:18:32 Running Dag: dag_1463523512622_0001_1
2016-05-17 22:18:32,822 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1463523512622_0001_1][Event:DAG_SUBMITTED]: dagID=dag_1463523512622_0001_1, submitTime=1463523512768
2016-05-17 22:18:32,823 INFO impl.VertexImpl (VertexImpl.java:setAdditionalOutputs(4033)) - setting additional outputs for vertex DAF899C218864FA991BD8131DDD089DB
2016-05-17 22:18:32,824 INFO impl.DAGImpl (DAGImpl.java:assignDAGScheduler(1381)) - Using DAG Scheduler: org.apache.tez.dag.app.dag.impl.DAGSchedulerNaturalOrderControlled
2016-05-17 22:18:32,824 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event DAG_SUBMITTED to history file
2016-05-17 22:18:32,826 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event DAG_INITIALIZED to history file
2016-05-17 22:18:32,826 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1463523512622_0001_1][Event:DAG_INITIALIZED]: dagID=dag_1463523512622_0001_1, initTime=1463523512823
2016-05-17 22:18:32,826 INFO impl.DAGImpl (DAGImpl.java:handle(973)) - dag_1463523512622_0001_1 transitioned from NEW to INITED due to event DAG_INIT
2016-05-17 22:18:32,827 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1463523512622_0001_1][Event:DAG_STARTED]: dagID=dag_1463523512622_0001_1, startTime=1463523512827
2016-05-17 22:18:32,827 INFO impl.DAGImpl (DAGImpl.java:transition(1614)) - Added additional resources : [[]] to classpath
2016-05-17 22:18:32,827 INFO impl.DAGImpl (DAGImpl.java:handle(973)) - dag_1463523512622_0001_1 transitioned from INITED to RUNNING due to event DAG_START
2016-05-17 22:18:32,828 INFO impl.VertexImpl (VertexImpl.java:setupVertex(2068)) - Root Inputs exist for Vertex: D65D4255FC664CD1A782B83C7638434D : {C772F8C9A29F4FECB6CBAA6E87DBBDAC={InputName=C772F8C9A29F4FECB6CBAA6E87DBBDAC}, {Descriptor=ClassName=org.apache.tez.mapreduce.input.MRInput, hasPayload=true}, {ControllerDescriptor=ClassName=org.apache.tez.mapreduce.common.MRInputAMSplitGenerator, hasPayload=false}}
2016-05-17 22:18:32,828 INFO impl.VertexImpl (VertexImpl.java:setupVertex(2078)) - Starting root input initializer for input: C772F8C9A29F4FECB6CBAA6E87DBBDAC, with class: [org.apache.tez.mapreduce.common.MRInputAMSplitGenerator]
2016-05-17 22:18:32,828 INFO impl.VertexImpl (VertexImpl.java:assignVertexManager(2191)) - Setting vertexManager to RootInputVertexManager for vertex_1463523512622_0001_1_00 [D65D4255FC664CD1A782B83C7638434D]
2016-05-17 22:18:32,828 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(2983)) - Num tasks is -1. Expecting VertexManager/InputInitializers/1-1 split to set #tasks for the vertex vertex_1463523512622_0001_1_00 [D65D4255FC664CD1A782B83C7638434D]
2016-05-17 22:18:32,828 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(2987)) - Vertex will initialize from input initializer. vertex_1463523512622_0001_1_00 [D65D4255FC664CD1A782B83C7638434D]
2016-05-17 22:18:32,829 INFO impl.VertexImpl (VertexImpl.java:setupInputInitializerManager(3933)) - Vertex will initialize via inputInitializers vertex_1463523512622_0001_1_00 [D65D4255FC664CD1A782B83C7638434D]. Starting root input initializers: 1
2016-05-17 22:18:32,829 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event DAG_STARTED to history file
2016-05-17 22:18:32,830 INFO impl.VertexImpl (VertexImpl.java:handle(1564)) - vertex_1463523512622_0001_1_00 [D65D4255FC664CD1A782B83C7638434D] transitioned from NEW to INITIALIZING due to event V_INIT
2016-05-17 22:18:32,830 INFO impl.VertexImpl (VertexImpl.java:assignVertexManager(2203)) - Setting vertexManager to ShuffleVertexManager for vertex_1463523512622_0001_1_01 [DAF899C218864FA991BD8131DDD089DB]
2016-05-17 22:18:32,834 INFO dag.RootInputInitializerManager (RootInputInitializerManager.java:run(242)) - Starting InputInitializer for Input: C772F8C9A29F4FECB6CBAA6E87DBBDAC on vertex vertex_1463523512622_0001_1_00 [D65D4255FC664CD1A782B83C7638434D]
2016-05-17 22:18:32,836 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:initialize(696)) - Shuffle Vertex Manager: settings minFrac:0.25 maxFrac:0.75 auto:false desiredTaskIput:104857600 minTasks:1
2016-05-17 22:18:32,836 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(3014)) - Creating 1 tasks for vertex: vertex_1463523512622_0001_1_01 [DAF899C218864FA991BD8131DDD089DB]
2016-05-17 22:18:32,836 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(3026)) - Directly initializing vertex: vertex_1463523512622_0001_1_01 [DAF899C218864FA991BD8131DDD089DB]
2016-05-17 22:18:32,836 INFO impl.VertexImpl (VertexImpl.java:initializeCommitters(1915)) - Invoking committer inits for vertex, vertexId=vertex_1463523512622_0001_1_01 [DAF899C218864FA991BD8131DDD089DB]
2016-05-17 22:18:32,837 INFO impl.VertexImpl (VertexImpl.java:initializeCommitters(1927)) - Instantiating committer for output=08300649FE3340E28A95D849CC8CB7E3, vertexId=vertex_1463523512622_0001_1_01 [DAF899C218864FA991BD8131DDD089DB], committerClass=org.apache.tez.mapreduce.committer.MROutputCommitter
2016-05-17 22:18:32,837 INFO impl.VertexImpl (VertexImpl.java:run(1945)) - Invoking committer init for output=08300649FE3340E28A95D849CC8CB7E3, vertexId=vertex_1463523512622_0001_1_01 [DAF899C218864FA991BD8131DDD089DB]
2016-05-17 22:18:32,838 INFO common.MRInputAMSplitGenerator (MRInputAMSplitGenerator.java:initialize(92)) - Emitting serialized splits: true
2016-05-17 22:18:32,838 INFO client.TezClient (TezClient.java:submitDAGSession(478)) - Submitted dag to TezSession, sessionName=(1/1) ...testSimpleBuffer2/simple2, applicationId=application_1463523512622_0001, dagName=[5F24C8AC118845E5B6CBE3B6796D5655/DBF10C5DC60F4509983187E52CD42569] test/(1/1) ...testSimpleBuffer2/simple2
2016-05-17 22:18:32,840 INFO common.MRInputAMSplitGenerator (MRInputAMSplitGenerator.java:initialize(110)) - Input C772F8C9A29F4FECB6CBAA6E87DBBDAC asking for 1 tasks. Headroom: 672 Task Resource: 1024 waves: 1.7
2016-05-17 22:18:32,840 INFO committer.MROutputCommitter (MROutputCommitter.java:getOutputCommitter(145)) - OutputCommitter for outputName=08300649FE3340E28A95D849CC8CB7E3, vertexName=DAF899C218864FA991BD8131DDD089DB, outputCommitterClass=org.apache.hadoop.mapred.FileOutputCommitter
2016-05-17 22:18:32,842 INFO common.MRInputAMSplitGenerator (MRInputAMSplitGenerator.java:initialize(121)) - Grouping input splits
2016-05-17 22:18:32,843 INFO hadoop.MRInputHelpers (MRInputHelpers.java:generateInputSplitsToMem(294)) - Generating mapred api input splits
2016-05-17 22:18:32,842 INFO impl.VertexImpl (VertexImpl.java:run(1949)) - Invoking committer setup for output=08300649FE3340E28A95D849CC8CB7E3, vertexId=vertex_1463523512622_0001_1_01 [DAF899C218864FA991BD8131DDD089DB]
2016-05-17 22:18:32,847 INFO hadoop.MRInputHelpers (MRInputHelpers.java:generateInputSplitsToMem(300)) - NumSplits: 1, SerializedSize: 396
2016-05-17 22:18:32,848 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1463523512622_0001_1][Event:VERTEX_INITIALIZED]: vertexName=DAF899C218864FA991BD8131DDD089DB, vertexId=vertex_1463523512622_0001_1_01, initRequestedTime=1463523512830, initedTime=1463523512846, numTasks=1, processorName=cascading.flow.tez.FlowProcessor, additionalInputsCount=0
2016-05-17 22:18:32,849 INFO impl.VertexImpl (VertexImpl.java:handle(1564)) - vertex_1463523512622_0001_1_01 [DAF899C218864FA991BD8131DDD089DB] transitioned from NEW to INITED due to event V_INIT
2016-05-17 22:18:32,850 INFO dag.RootInputInitializerManager (RootInputInitializerManager.java:onSuccess(272)) - Succeeded InputInitializer for Input: C772F8C9A29F4FECB6CBAA6E87DBBDAC on vertex vertex_1463523512622_0001_1_00 [D65D4255FC664CD1A782B83C7638434D]
2016-05-17 22:18:32,848 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event VERTEX_INITIALIZED to history file
2016-05-17 22:18:32,850 INFO impl.VertexImpl (VertexImpl.java:canInitVertex(3195)) - Cannot init vertex: vertex_1463523512622_0001_1_00 [D65D4255FC664CD1A782B83C7638434D] numTasks: -1 numUnitializedEdges: 0 numInitializedInputs: 0 initWaitsForRootInitializers: true
2016-05-17 22:18:32,851 INFO impl.VertexImpl (VertexImpl.java:setParallelism(1374)) - Got updated RootInputsSpecs: {C772F8C9A29F4FECB6CBAA6E87DBBDAC=forAllWorkUnits=true, update=[1]}
2016-05-17 22:18:32,852 INFO impl.VertexImpl (VertexImpl.java:setParallelism(1393)) - Vertex vertex_1463523512622_0001_1_00 [D65D4255FC664CD1A782B83C7638434D] parallelism set to 1
2016-05-17 22:18:32,852 INFO impl.VertexImpl (VertexImpl.java:canInitVertex(3195)) - Cannot init vertex: vertex_1463523512622_0001_1_00 [D65D4255FC664CD1A782B83C7638434D] numTasks: 1 numUnitializedEdges: 0 numInitializedInputs: 0 initWaitsForRootInitializers: true
2016-05-17 22:18:32,853 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1463523512622_0001_1][Event:VERTEX_DATA_MOVEMENT_EVENTS_GENERATED]: vertexId=vertex_1463523512622_0001_1_00, eventCount=1
2016-05-17 22:18:32,856 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event VERTEX_INITIALIZED to history file
2016-05-17 22:18:32,857 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1463523512622_0001_1][Event:VERTEX_INITIALIZED]: vertexName=D65D4255FC664CD1A782B83C7638434D, vertexId=vertex_1463523512622_0001_1_00, initRequestedTime=1463523512828, initedTime=1463523512855, numTasks=1, processorName=cascading.flow.tez.FlowProcessor, additionalInputsCount=1
2016-05-17 22:18:32,857 INFO impl.VertexImpl (VertexImpl.java:startIfPossible(3061)) - Triggering start event for vertex: vertex_1463523512622_0001_1_00 [D65D4255FC664CD1A782B83C7638434D] with distanceFromRoot: 0
2016-05-17 22:18:32,857 INFO impl.VertexImpl (VertexImpl.java:handle(1564)) - vertex_1463523512622_0001_1_00 [D65D4255FC664CD1A782B83C7638434D] transitioned from INITIALIZING to INITED due to event V_ROOT_INPUT_INITIALIZED
2016-05-17 22:18:32,858 INFO impl.ImmediateStartVertexManager (ImmediateStartVertexManager.java:scheduleTasks(99)) - Starting 1 in D65D4255FC664CD1A782B83C7638434D
2016-05-17 22:18:32,859 INFO impl.VertexImpl (VertexImpl.java:scheduleTasks(1250)) - Routing pending task events for vertex: vertex_1463523512622_0001_1_00 [D65D4255FC664CD1A782B83C7638434D]
2016-05-17 22:18:32,860 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event VERTEX_STARTED to history file
2016-05-17 22:18:32,860 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1463523512622_0001_1][Event:VERTEX_STARTED]: vertexId=vertex_1463523512622_0001_1_00, startRequestedTime=1463523512858, startedTime=1463523512858
2016-05-17 22:18:32,861 INFO impl.VertexImpl (VertexImpl.java:handle(1564)) - vertex_1463523512622_0001_1_00 [D65D4255FC664CD1A782B83C7638434D] transitioned from INITED to RUNNING due to event V_START
2016-05-17 22:18:32,862 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [test] submitted tez dag to app master: application_1463523512622_0001, with dag id: dag_1463523512622_0001_1
2016-05-17 22:18:32,862 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event TASK_STARTED to history file
2016-05-17 22:18:32,863 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1463523512622_0001_1][Event:TASK_STARTED]: vertexName=D65D4255FC664CD1A782B83C7638434D, taskId=task_1463523512622_0001_1_00_000000, scheduledTime=1463523512862, launchTime=1463523512862
2016-05-17 22:18:32,863 INFO impl.TaskImpl (TaskImpl.java:handle(862)) - task_1463523512622_0001_1_00_000000 Task Transitioned from NEW to SCHEDULED due to event T_SCHEDULE
2016-05-17 22:18:32,864 INFO impl.VertexImpl (VertexImpl.java:transition(3155)) - Source vertex started: vertex_1463523512622_0001_1_00 for vertex: vertex_1463523512622_0001_1_01 [DAF899C218864FA991BD8131DDD089DB] numStartedSources: 1 numSources: 1
2016-05-17 22:18:32,864 INFO impl.VertexImpl (VertexImpl.java:startIfPossible(3061)) - Triggering start event for vertex: vertex_1463523512622_0001_1_01 [DAF899C218864FA991BD8131DDD089DB] with distanceFromRoot: 1
2016-05-17 22:18:32,864 INFO impl.DAGSchedulerNaturalOrderControlled (DAGSchedulerNaturalOrderControlled.java:scheduleTask(111)) - Scheduled vertex: vertex_1463523512622_0001_1_00 [D65D4255FC664CD1A782B83C7638434D]
2016-05-17 22:18:32,865 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:handleVertexStateUpdate(720)) - Received configured notification : CONFIGURED for vertex: D65D4255FC664CD1A782B83C7638434D in vertex: DAF899C218864FA991BD8131DDD089DB
2016-05-17 22:18:32,865 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:onVertexStarted(355)) - OnVertexStarted vertex: DAF899C218864FA991BD8131DDD089DB with 1 source tasks and 1 pending tasks
2016-05-17 22:18:32,865 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event VERTEX_STARTED to history file
2016-05-17 22:18:32,866 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1463523512622_0001_1][Event:VERTEX_STARTED]: vertexId=vertex_1463523512622_0001_1_01, startRequestedTime=1463523512864, startedTime=1463523512864
2016-05-17 22:18:32,866 INFO impl.VertexImpl (VertexImpl.java:handle(1564)) - vertex_1463523512622_0001_1_01 [DAF899C218864FA991BD8131DDD089DB] transitioned from INITED to RUNNING due to event V_START
2016-05-17 22:18:32,867 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:transition(1014)) - remoteTaskSpec:DAGName : [5F24C8AC118845E5B6CBE3B6796D5655/DBF10C5DC60F4509983187E52CD42569] test/(1/1) ...testSimpleBuffer2/simple2, VertexName: D65D4255FC664CD1A782B83C7638434D, VertexParallelism: 1, TaskAttemptID:attempt_1463523512622_0001_1_00_000000_0, processorName=cascading.flow.tez.FlowProcessor, inputSpecListSize=1, outputSpecListSize=1, inputSpecList=[{{ sourceVertexName=C772F8C9A29F4FECB6CBAA6E87DBBDAC, physicalEdgeCount=1, inputClassName=org.apache.tez.mapreduce.input.MRInput }}, ], outputSpecList=[{{ destinationVertexName=DAF899C218864FA991BD8131DDD089DB, physicalEdgeCount=1, outputClassName=org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput }}, ]
2016-05-17 22:18:32,867 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:handle(692)) - attempt_1463523512622_0001_1_00_000000_0 TaskAttempt Transitioned from NEW to START_WAIT due to event TA_SCHEDULE
2016-05-17 22:18:32,867 INFO rm.TaskSchedulerEventHandler (TaskSchedulerEventHandler.java:handleEvent(149)) - Processing the event EventType: S_TA_LAUNCH_REQUEST
2016-05-17 22:18:32,869 INFO node.AMNodeTracker (AMNodeTracker.java:nodeSeen(91)) - Adding new node: 127.0.0.1:0
2016-05-17 22:18:32,870 INFO container.AMContainerHelpers (AMContainerHelpers.java:createCommonContainerLaunchContext(112)) - Adding #1 tokens and #0 secret keys for NM use for launching container
2016-05-17 22:18:32,871 INFO container.AMContainerHelpers (AMContainerHelpers.java:createCommonContainerLaunchContext(122)) - Putting shuffle token in serviceData
2016-05-17 22:18:32,872 INFO container.AMContainerImpl (AMContainerImpl.java:transition(521)) - Sending Launch Request for Container with id: container_1463523512622_0001_00_000001
2016-05-17 22:18:32,872 INFO container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1463523512622_0001_00_000001 transitioned from ALLOCATED to LAUNCHING via event C_LAUNCH_REQUEST
2016-05-17 22:18:32,874 INFO task.TezChild (TezChild.java:newTezChild(432)) - PID, containerIdentifier: null, container_1463523512622_0001_00_000001
2016-05-17 22:18:32,874 WARN impl.MetricsSystemImpl (MetricsSystemImpl.java:init(145)) - DAGAppMaster metrics system already initialized!
2016-05-17 22:18:32,877 INFO container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1463523512622_0001_00_000001 transitioned from LAUNCHING to IDLE via event C_LAUNCHED
2016-05-17 22:18:32,877 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event CONTAINER_LAUNCHED to history file
2016-05-17 22:18:32,877 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1463523512622_0001_1][Event:CONTAINER_LAUNCHED]: containerId=container_1463523512622_0001_00_000001, launchTime=1463523512877
2016-05-17 22:18:32,879 INFO task.ContainerReporter (ContainerReporter.java:call(55)) - Attempting to fetch new task for container container_1463523512622_0001_00_000001
2016-05-17 22:18:32,880 INFO container.AMContainerImpl (AMContainerImpl.java:transition(827)) - Assigned taskAttempt + [attempt_1463523512622_0001_1_00_000000_0] to container: [container_1463523512622_0001_00_000001]
2016-05-17 22:18:32,880 INFO container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1463523512622_0001_00_000001 transitioned from IDLE to RUNNING via event C_PULL_TA
2016-05-17 22:18:32,880 INFO app.TaskAttemptListenerImpTezDag (TaskAttemptListenerImpTezDag.java:getTask(240)) - Container with id: container_1463523512622_0001_00_000001 given task: attempt_1463523512622_0001_1_00_000000_0
2016-05-17 22:18:32,881 INFO task.ContainerReporter (ContainerReporter.java:call(65)) - Got TaskUpdate for containerId= container_1463523512622_0001_00_000001: 0 ms after starting to poll. TaskInfo: shouldDie: false, currentTaskAttemptId: attempt_1463523512622_0001_1_00_000000_0
2016-05-17 22:18:32,881 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:transition(1146)) - TaskAttempt: [attempt_1463523512622_0001_1_00_000000_0] started. Is using containerId: [container_1463523512622_0001_00_000001] on NM: [127.0.0.1:0]
2016-05-17 22:18:32,881 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1463523512622_0001_1][Event:TASK_ATTEMPT_STARTED]: vertexName=D65D4255FC664CD1A782B83C7638434D, taskAttemptId=attempt_1463523512622_0001_1_00_000000_0, startTime=1463523512881, containerId=container_1463523512622_0001_00_000001, nodeId=127.0.0.1:0, inProgressLogs=127.0.0.1:0/node/containerlogs/container_1463523512622_0001_00_000001/teamcity, completedLogs=
2016-05-17 22:18:32,882 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event TASK_ATTEMPT_STARTED to history file
2016-05-17 22:18:32,882 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:handle(692)) - attempt_1463523512622_0001_1_00_000000_0 TaskAttempt Transitioned from START_WAIT to RUNNING due to event TA_STARTED_REMOTELY
2016-05-17 22:18:32,882 INFO impl.TaskImpl (TaskImpl.java:handle(862)) - task_1463523512622_0001_1_00_000000 Task Transitioned from SCHEDULED to RUNNING due to event T_ATTEMPT_LAUNCHED
2016-05-17 22:18:32 Starting to run new task attempt: attempt_1463523512622_0001_1_00_000000_0
2016-05-17 22:18:32,883 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(164)) - Redirecting log file based on addend: attempt_1463523512622_0001_1_00_000000_0
2016-05-17 22:18:32,883 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(179)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-17 22:18:32,883 INFO task.TezChild (TezChild.java:handleNewTaskCredentials(273)) - Refreshing UGI since Credentials have changed
2016-05-17 22:18:32,883 INFO task.TezChild (TezChild.java:handleNewTaskCredentials(276)) - Credentials : #Tokens=1, #SecretKeys=0
2016-05-17 22:18:32,884 INFO task.TezChild (TezChild.java:handleNewTaskLocalResources(302)) - Localizing additional local resources for Task : {}
2016-05-17 22:18:32,884 INFO task.TezChild (TezChild.java:handleNewTaskLocalResources(312)) - Done localizing additional resources
2016-05-17 22:18:32,884 INFO metrics.TaskCounterUpdater (TaskCounterUpdater.java:initResourceCalculatorPlugin(148)) - Using ResourceCalculatorProcessTree : [ ]
2016-05-17 22:18:32,885 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:<init>(149)) - Initializing LogicalIOProcessorRuntimeTask with TaskSpec: DAGName : [5F24C8AC118845E5B6CBE3B6796D5655/DBF10C5DC60F4509983187E52CD42569] test/(1/1) ...testSimpleBuffer2/simple2, VertexName: D65D4255FC664CD1A782B83C7638434D, VertexParallelism: 1, TaskAttemptID:attempt_1463523512622_0001_1_00_000000_0, processorName=cascading.flow.tez.FlowProcessor, inputSpecListSize=1, outputSpecListSize=1, inputSpecList=[{{ sourceVertexName=C772F8C9A29F4FECB6CBAA6E87DBBDAC, physicalEdgeCount=1, inputClassName=org.apache.tez.mapreduce.input.MRInput }}, ], outputSpecList=[{{ destinationVertexName=DAF899C218864FA991BD8131DDD089DB, physicalEdgeCount=1, outputClassName=org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput }}, ]
2016-05-17 22:18:32,885 INFO resources.MemoryDistributor (MemoryDistributor.java:<init>(85)) - InitialMemoryDistributor (isEnabled=true) invoked with: numInputs=1, numOutputs=1, JVM.maxFree=704643072
2016-05-17 22:18:32,886 INFO task.TezTaskRunner (TezTaskRunner.java:run(172)) - Initializing task, taskAttemptId=attempt_1463523512622_0001_1_00_000000_0
2016-05-17 22:18:32,886 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(191)) - Initializing LogicalProcessorIORuntimeTask
2016-05-17 22:18:32,886 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(195)) - Creating processor, processorClassName=cascading.flow.tez.FlowProcessor
2016-05-17 22:18:32,887 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(389)) - Initializing Input using InputSpec: { sourceVertexName=C772F8C9A29F4FECB6CBAA6E87DBBDAC, physicalEdgeCount=1, inputClassName=org.apache.tez.mapreduce.input.MRInput }
2016-05-17 22:18:32,888 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:createInput(537)) - Creating Input
2016-05-17 22:18:32,889 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(397)) - Initializing Input with src edge: C772F8C9A29F4FECB6CBAA6E87DBBDAC
2016-05-17 22:18:32,889 INFO resources.MemoryDistributor$RequestorInfo (MemoryDistributor.java:<init>(233)) - Received request: 0, type: INPUT, componentVertexName: C772F8C9A29F4FECB6CBAA6E87DBBDAC
2016-05-17 22:18:32,892 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initializeLogicalIOProcessor(488)) - Initializing processor, processorClassName=cascading.flow.tez.FlowProcessor
2016-05-17 22:18:32,895 INFO impl.TaskImpl (TaskImpl.java:getTaskAttemptTezEvents(481)) - TaskAttempt:attempt_1463523512622_0001_1_00_000000_0 sent events: (0-1)
2016-05-17 22:18:32,896 INFO input.MRInput (MRInput.java:initialize(440)) - Using New mapreduce API: false, split information via event: true
2016-05-17 22:18:32,896 INFO input.MRInput (MRInput.java:initializeInternal(493)) - Initialzed MRInput: C772F8C9A29F4FECB6CBAA6E87DBBDAC
2016-05-17 22:18:32,896 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(403)) - Initialized Input with src edge: C772F8C9A29F4FECB6CBAA6E87DBBDAC
2016-05-17 22:18:32,897 INFO tez.FlowProcessor (FlowProcessor.java:initialize(88)) - cascading version: 3.1.0
2016-05-17 22:18:32,899 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(438)) - Initializing Output using OutputSpec: { destinationVertexName=DAF899C218864FA991BD8131DDD089DB, physicalEdgeCount=1, outputClassName=org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput }
2016-05-17 22:18:32,899 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:createOutput(565)) - Creating Output
2016-05-17 22:18:32,902 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(446)) - Initializing Output with dest edge: DAF899C218864FA991BD8131DDD089DB
2016-05-17 22:18:32,905 INFO impl.ExternalSorter (ExternalSorter.java:getInitialMemoryRequirement(293)) - Requested SortBufferSize (tez.runtime.io.sort.mb): 100
2016-05-17 22:18:32,906 INFO resources.MemoryDistributor$RequestorInfo (MemoryDistributor.java:<init>(233)) - Received request: 104857600, type: OUTPUT, componentVertexName: DAF899C218864FA991BD8131DDD089DB
2016-05-17 22:18:32,907 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(452)) - Initialized Output with dest edge: DAF899C218864FA991BD8131DDD089DB
2016-05-17 22:18:32,908 INFO tez.FlowProcessor (FlowProcessor.java:initialize(94)) - flow node id: D65D4255FC664CD1A782B83C7638434D, ordinal: 0
2016-05-17 22:18:32,908 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: D65D4255FC664CD1A782B83C7638434D, mem on start (mb), free: 199, total: 528, max: 672
2016-05-17 22:18:32,909 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initializeLogicalIOProcessor(491)) - Initialized processor, processorClassName=cascading.flow.tez.FlowProcessor
2016-05-17 22:18:32,909 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(220)) - Waiting for 2 initializers to finish
2016-05-17 22:18:32,909 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(220)) - Waiting for 1 initializers to finish
2016-05-17 22:18:32,909 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(233)) - All initializers finished
2016-05-17 22:18:32,909 INFO resources.MemoryDistributor (MemoryDistributor.java:makeInitialAllocations(124)) - Using Allocator class: org.apache.tez.runtime.library.resources.WeightedScalingMemoryDistributor
2016-05-17 22:18:32,909 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:getRequestTypeForClass(197)) - Falling back to RequestType.OTHER for class: org.apache.tez.mapreduce.input.MRInput
2016-05-17 22:18:32,910 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:initialProcessMemoryRequestContext(171)) - ScaleFactor: 1, for type: OTHER
2016-05-17 22:18:32,910 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:initialProcessMemoryRequestContext(171)) - ScaleFactor: 12, for type: SORTED_OUTPUT
2016-05-17 22:18:32,910 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:computeReservedFraction(262)) - InitialReservationFraction=0.3, AdditionalReservationFractionForIOs=0.03, finalReserveFractionUsed=0.32999999999999996
2016-05-17 22:18:32,911 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:assignMemory(126)) - Scaling Requests. NumRequests: 2, numScaledRequests: 13, TotalRequested: 104857600, TotalRequestedScaled: 9.679163076923077E7, TotalJVMHeap: 704643072, TotalAvailable: 472110858, TotalRequested/TotalJVMHeap:0.15
2016-05-17 22:18:32,911 INFO resources.MemoryDistributor (MemoryDistributor.java:makeInitialAllocations(138)) - Informing: INPUT, C772F8C9A29F4FECB6CBAA6E87DBBDAC, org.apache.tez.mapreduce.input.MRInput: requested=0, allocated=0
2016-05-17 22:18:32,911 INFO resources.MemoryDistributor (MemoryDistributor.java:makeInitialAllocations(138)) - Informing: OUTPUT, DAF899C218864FA991BD8131DDD089DB, org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput: requested=104857600, allocated=104857600
2016-05-17 22:18:32,912 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(255)) - Starting Inputs/Outputs
2016-05-17 22:18:32,912 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(419)) - Starting Input with src edge: C772F8C9A29F4FECB6CBAA6E87DBBDAC
2016-05-17 22:18:32,912 INFO input.MRInput (MRInput.java:start(450)) - MRInput setup to received events1
2016-05-17 22:18:32,912 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(421)) - Started Input with src edge: C772F8C9A29F4FECB6CBAA6E87DBBDAC
2016-05-17 22:18:32,913 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(268)) - Input: C772F8C9A29F4FECB6CBAA6E87DBBDAC being auto started by the framework. Subsequent instances will not be auto-started
2016-05-17 22:18:32,913 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(289)) - Num IOs determined for AutoStart: 1
2016-05-17 22:18:32,913 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(291)) - Waiting for 1 IOs to start
2016-05-17 22:18:32,914 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(304)) - AutoStartComplete
2016-05-17 22:18:32,914 INFO task.TezTaskRunner (TezTaskRunner.java:run(175)) - Running task, taskAttemptId=attempt_1463523512622_0001_1_00_000000_0
2016-05-17 22:18:32,920 INFO input.MRInput (MRInput.java:initFromEventInternal(619)) - Initializing RecordReader from event
2016-05-17 22:18:32,921 INFO graph.Hadoop2TezStreamGraph (Hadoop2TezStreamGraph.java:buildGraph(151)) - using streamed source: Hfs["TextLine[['offset', 'line' | long, String]->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.5.x-tez-0.6.x/build/classes/test/data/lhs.txt"]
2016-05-17 22:18:32,922 INFO input.MRInput (MRInput.java:initFromEventInternal(630)) - Split Details -> SplitClass: org.apache.hadoop.mapred.split.TezGroupedSplit, OldSplit: org.apache.hadoop.mapred.split.TezGroupedSplit@7e1b4e9d
2016-05-17 22:18:32,923 INFO tez.FlowProcessor (FlowProcessor.java:run(121)) - sourcing from: Hfs["TextLine[['offset', 'line' | long, String]->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.5.x-tez-0.6.x/build/classes/test/data/lhs.txt"] streamed: true, id: C772F8C9A29F4FECB6CBAA6E87DBBDAC
2016-05-17 22:18:32,923 INFO tez.FlowProcessor (FlowProcessor.java:run(124)) - sinking to: GroupBy(test)[by:[{1}:'num']], id: 165EBC44B5354F26A0228F229AE27CCF
2016-05-17 22:18:32,923 INFO element.TezGroupGate (TezGroupGate.java:prepare(98)) - calling OrderedPartitionedKVOutput#start() on: GroupBy(test)[by:[{1}:'num']] 165EBC44B5354F26A0228F229AE27CCF
2016-05-17 22:18:32,924 INFO lib.MRReaderMapred (MRReaderMapred.java:setIncrementalConfigParams(163)) - Processing split: org.apache.hadoop.mapred.split.TezGroupedSplit@7e1b4e9d
2016-05-17 22:18:32,924 INFO input.MRInput (MRInput.java:initFromEventInternal(634)) - Initialized RecordReader from event
2016-05-17 22:18:32,926 INFO input.MRInput (MRInput.java:processSplitEvent(586)) - Notifying on RecordReader Initialized
2016-05-17 22:18:32,926 INFO impl.ExternalSorter (ExternalSorter.java:<init>(167)) - keySerializer=cascading.tuple.hadoop.io.KeyTupleSerializer@124ca3bd; valueSerializer=cascading.tuple.hadoop.io.ValueTupleSerializer@6591d6ad; comparator=cascading.tuple.hadoop.util.TupleComparator@49a52bf3; conf=cascading.tuple.hadoop.TupleSerialization,org.apache.hadoop.io.serializer.WritableSerialization,org.apache.hadoop.io.serializer.avro.AvroSpecificSerialization,org.apache.hadoop.io.serializer.avro.AvroReflectSerialization
2016-05-17 22:18:32,927 INFO impl.ExternalSorter (ExternalSorter.java:<init>(224)) - Instantiating Partitioner: [cascading.tuple.tez.util.TuplePartitioner]
2016-05-17 22:18:32,927 INFO common.TezRuntimeUtils (TezRuntimeUtils.java:instantiatePartitioner(109)) - Using partitioner class: cascading.tuple.tez.util.TuplePartitioner
2016-05-17 22:18:32,928 INFO common.TezRuntimeUtils (TezRuntimeUtils.java:instantiateCombiner(65)) - No combiner specified via tez.runtime.combiner.class. Combiner will not be used
2016-05-17 22:18:33,089 INFO dflt.DefaultSorter (DefaultSorter.java:setEquator(328)) - (EQUATOR) 0 kvi 26214396(104857584)
2016-05-17 22:18:33,089 INFO dflt.DefaultSorter (DefaultSorter.java:<init>(150)) - tez.runtime.io.sort.mb: 100
2016-05-17 22:18:33,089 INFO dflt.DefaultSorter (DefaultSorter.java:<init>(151)) - soft limit at 83886080
2016-05-17 22:18:33,090 INFO dflt.DefaultSorter (DefaultSorter.java:<init>(152)) - bufstart = 0; bufvoid = 104857600
2016-05-17 22:18:33,090 INFO dflt.DefaultSorter (DefaultSorter.java:<init>(153)) - kvstart = 26214396; length = 6553600
2016-05-17 22:18:33,091 INFO element.TezSourceStage (TezSourceStage.java:prepare(63)) - calling MRInput#start() on: Hfs["TextLine[['offset', 'line' | long, String]->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.5.x-tez-0.6.x/build/classes/test/data/lhs.txt"]
2016-05-17 22:18:33,091 INFO input.MRInput (MRInput.java:start(450)) - MRInput setup to received events1
2016-05-17 22:18:33,091 INFO tez.FlowProcessor (FlowProcessor.java:waitForInputsReady(202)) - flow node id: D65D4255FC664CD1A782B83C7638434D, all 1 inputs ready in: 00:00:00.000
2016-05-17 22:18:33,095 INFO task.TezTaskRunner (TezTaskRunner.java:run(177)) - Closing task, taskAttemptId=attempt_1463523512622_0001_1_00_000000_0
2016-05-17 22:18:33,096 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: D65D4255FC664CD1A782B83C7638434D, mem on close (mb), free: 395, total: 515, max: 672
2016-05-17 22:18:33,096 INFO dflt.DefaultSorter (DefaultSorter.java:flush(589)) - Starting flush of map output
2016-05-17 22:18:33,096 INFO dflt.DefaultSorter (DefaultSorter.java:flush(607)) - Sorting & Spilling map output
2016-05-17 22:18:33,097 INFO dflt.DefaultSorter (DefaultSorter.java:flush(608)) - bufstart = 0; bufend = 195; bufvoid = 104857600
2016-05-17 22:18:33,097 INFO dflt.DefaultSorter (DefaultSorter.java:flush(610)) - kvstart = 26214396(104857584); kvend = 26214348(104857392); length = 49/6553600
2016-05-17 22:18:33,102 INFO dflt.DefaultSorter (DefaultSorter.java:spill(829)) - Finished spill 0
2016-05-17 22:18:33 Completed running task attempt: attempt_1463523512622_0001_1_00_000000_0
2016-05-17 22:18:33,107 INFO task.TezTaskRunner (TezTaskRunner.java:run(181)) - Task completed, taskAttemptId=attempt_1463523512622_0001_1_00_000000_0, fatalErrorOccurred=false
2016-05-17 22:18:33,113 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1463523512622_0001_1][Event:VERTEX_DATA_MOVEMENT_EVENTS_GENERATED]: vertexId=vertex_1463523512622_0001_1_00, eventCount=1
2016-05-17 22:18:33,114 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:cleanup(712)) - Final Counters : Counters: 27 [[File System Counters FILE_BYTES_READ=110, FILE_BYTES_WRITTEN=207, FILE_READ_OPS=0, FILE_LARGE_READ_OPS=0, FILE_WRITE_OPS=0][org.apache.tez.common.counters.TaskCounter SPILLED_RECORDS=13, GC_TIME_MILLIS=146, CPU_MILLISECONDS=0, PHYSICAL_MEMORY_BYTES=0, VIRTUAL_MEMORY_BYTES=0, COMMITTED_HEAP_BYTES=540540928, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=13, OUTPUT_BYTES=195, OUTPUT_BYTES_WITH_OVERHEAD=171, OUTPUT_BYTES_PHYSICAL=175, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=0, ADDITIONAL_SPILL_COUNT=0][cascading.flow.SliceCounters Process_Begin_Time=1463523513091, Process_Duration=4, Process_End_Time=1463523513095, Read_Duration=2, Tuples_Read=13, Tuples_Written=13, Write_Duration=0][cascading.flow.StepCounters Tuples_Read=13]]
2016-05-17 22:18:33,115 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event TASK_ATTEMPT_FINISHED to history file
2016-05-17 22:18:33,114 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1463523512622_0001_1][Event:TASK_ATTEMPT_FINISHED]: vertexName=D65D4255FC664CD1A782B83C7638434D, taskAttemptId=attempt_1463523512622_0001_1_00_000000_0, startTime=1463523512881, finishTime=1463523513114, timeTaken=233, status=SUCCEEDED, errorEnum=, diagnostics=, counters=Counters: 28, org.apache.tez.common.counters.DAGCounter, RACK_LOCAL_TASKS=1, File System Counters, FILE_BYTES_READ=110, FILE_BYTES_WRITTEN=207, FILE_READ_OPS=0, FILE_LARGE_READ_OPS=0, FILE_WRITE_OPS=0, org.apache.tez.common.counters.TaskCounter, SPILLED_RECORDS=13, GC_TIME_MILLIS=146, CPU_MILLISECONDS=0, PHYSICAL_MEMORY_BYTES=0, VIRTUAL_MEMORY_BYTES=0, COMMITTED_HEAP_BYTES=540540928, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=13, OUTPUT_BYTES=195, OUTPUT_BYTES_WITH_OVERHEAD=171, OUTPUT_BYTES_PHYSICAL=175, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=0, ADDITIONAL_SPILL_COUNT=0, cascading.flow.SliceCounters, Process_Begin_Time=1463523513091, Process_Duration=4, Process_End_Time=1463523513095, Read_Duration=2, Tuples_Read=13, Tuples_Written=13, Write_Duration=0, cascading.flow.StepCounters, Tuples_Read=13
2016-05-17 22:18:33,115 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:handle(692)) - attempt_1463523512622_0001_1_00_000000_0 TaskAttempt Transitioned from RUNNING to SUCCEEDED due to event TA_DONE
2016-05-17 22:18:33,116 INFO impl.TaskImpl (TaskImpl.java:transition(1052)) - Task succeeded with attempt attempt_1463523512622_0001_1_00_000000_0
2016-05-17 22:18:33,117 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1463523512622_0001_1][Event:TASK_FINISHED]: vertexName=D65D4255FC664CD1A782B83C7638434D, taskId=task_1463523512622_0001_1_00_000000, startTime=1463523512881, finishTime=1463523513116, timeTaken=235, status=SUCCEEDED, successfulAttemptID=attempt_1463523512622_0001_1_00_000000_0, diagnostics=, counters=Counters: 28, org.apache.tez.common.counters.DAGCounter, RACK_LOCAL_TASKS=1, File System Counters, FILE_BYTES_READ=110, FILE_BYTES_WRITTEN=207, FILE_READ_OPS=0, FILE_LARGE_READ_OPS=0, FILE_WRITE_OPS=0, org.apache.tez.common.counters.TaskCounter, SPILLED_RECORDS=13, GC_TIME_MILLIS=146, CPU_MILLISECONDS=0, PHYSICAL_MEMORY_BYTES=0, VIRTUAL_MEMORY_BYTES=0, COMMITTED_HEAP_BYTES=540540928, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=13, OUTPUT_BYTES=195, OUTPUT_BYTES_WITH_OVERHEAD=171, OUTPUT_BYTES_PHYSICAL=175, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=0, ADDITIONAL_SPILL_COUNT=0, cascading.flow.SliceCounters, Process_Begin_Time=1463523513091, Process_Duration=4, Process_End_Time=1463523513095, Read_Duration=2, Tuples_Read=13, Tuples_Written=13, Write_Duration=0, cascading.flow.StepCounters, Tuples_Read=13
2016-05-17 22:18:33,117 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event TASK_FINISHED to history file
2016-05-17 22:18:33,117 INFO impl.TaskImpl (TaskImpl.java:handle(862)) - task_1463523512622_0001_1_00_000000 Task Transitioned from RUNNING to SUCCEEDED due to event T_ATTEMPT_SUCCEEDED
2016-05-17 22:18:33,117 INFO impl.VertexImpl (VertexImpl.java:transition(3536)) - Num completed Tasks for vertex_1463523512622_0001_1_00 [D65D4255FC664CD1A782B83C7638434D] : 1
2016-05-17 22:18:33,117 INFO impl.VertexImpl (VertexImpl.java:checkVertexForCompletion(1674)) - Vertex succeeded: vertex_1463523512622_0001_1_00 [D65D4255FC664CD1A782B83C7638434D]
2016-05-17 22:18:33,117 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(164)) - Redirecting log file based on addend:
2016-05-17 22:18:33,118 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(179)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-17 22:18:33,118 INFO rm.TaskSchedulerEventHandler (TaskSchedulerEventHandler.java:handleEvent(149)) - Processing the event EventType: S_TA_ENDED
2016-05-17 22:18:33,119 INFO task.ContainerReporter (ContainerReporter.java:call(55)) - Attempting to fetch new task for container container_1463523512622_0001_00_000001
2016-05-17 22:18:33,120 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1463523512622_0001_1][Event:VERTEX_FINISHED]: vertexName=D65D4255FC664CD1A782B83C7638434D, vertexId=vertex_1463523512622_0001_1_00, initRequestedTime=1463523512828, initedTime=1463523512855, startRequestedTime=1463523512858, startedTime=1463523512858, finishTime=1463523513117, timeTaken=259, status=SUCCEEDED, diagnostics=, counters=Counters: 28, org.apache.tez.common.counters.DAGCounter, RACK_LOCAL_TASKS=1, File System Counters, FILE_BYTES_READ=110, FILE_BYTES_WRITTEN=207, FILE_READ_OPS=0, FILE_LARGE_READ_OPS=0, FILE_WRITE_OPS=0, org.apache.tez.common.counters.TaskCounter, SPILLED_RECORDS=13, GC_TIME_MILLIS=146, CPU_MILLISECONDS=0, PHYSICAL_MEMORY_BYTES=0, VIRTUAL_MEMORY_BYTES=0, COMMITTED_HEAP_BYTES=540540928, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=13, OUTPUT_BYTES=195, OUTPUT_BYTES_WITH_OVERHEAD=171, OUTPUT_BYTES_PHYSICAL=175, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=0, ADDITIONAL_SPILL_COUNT=0, cascading.flow.SliceCounters, Process_Begin_Time=1463523513091, Process_Duration=4, Process_End_Time=1463523513095, Read_Duration=2, Tuples_Read=13, Tuples_Written=13, Write_Duration=0, cascading.flow.StepCounters, Tuples_Read=13, vertexStats=firstTaskStartTime=1463523512881, firstTasksToStart=[ task_1463523512622_0001_1_00_000000 ], lastTaskFinishTime=1463523513114, lastTasksToFinish=[ task_1463523512622_0001_1_00_000000 ], minTaskDuration=233, maxTaskDuration=233, avgTaskDuration=233.0, numSuccessfulTasks=1, shortestDurationTasks=[ task_1463523512622_0001_1_00_000000 ], longestDurationTasks=[ task_1463523512622_0001_1_00_000000 ], vertexTaskStats={numFailedTasks=0, numSucceededTasks=1, numKilledTaskAttempts=0, numKilledTasks=0, numFailedTaskAttempts=0, numCompletedTasks=1}
2016-05-17 22:18:33,120 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event VERTEX_FINISHED to history file
2016-05-17 22:18:33,120 INFO impl.VertexImpl (VertexImpl.java:handle(1564)) - vertex_1463523512622_0001_1_00 [D65D4255FC664CD1A782B83C7638434D] transitioned from RUNNING to SUCCEEDED due to event V_TASK_COMPLETED
2016-05-17 22:18:33,120 INFO impl.VertexImpl (VertexImpl.java:transition(3477)) - Source task attempt completed for vertex: vertex_1463523512622_0001_1_01 [DAF899C218864FA991BD8131DDD089DB] attempt: attempt_1463523512622_0001_1_00_000000_0 with state: SUCCEEDED vertexState: RUNNING
2016-05-17 22:18:33,121 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:schedulePendingTasks(608)) - All source tasks assigned. Ramping up 1 remaining tasks for vertex: DAF899C218864FA991BD8131DDD089DB
2016-05-17 22:18:33,121 INFO impl.VertexImpl (VertexImpl.java:scheduleTasks(1250)) - Routing pending task events for vertex: vertex_1463523512622_0001_1_01 [DAF899C218864FA991BD8131DDD089DB]
2016-05-17 22:18:33,121 INFO container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1463523512622_0001_00_000001 transitioned from RUNNING to IDLE via event C_TA_SUCCEEDED
2016-05-17 22:18:33,121 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1463523512622_0001_1][Event:CONTAINER_STOPPED]: containerId=container_1463523512622_0001_00_000001, stoppedTime=1463523513121, exitStatus=0
2016-05-17 22:18:33,121 INFO container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1463523512622_0001_00_000001 transitioned from IDLE to STOP_REQUESTED via event C_STOP_REQUEST
2016-05-17 22:18:33,122 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event CONTAINER_STOPPED to history file
2016-05-17 22:18:33,123 INFO impl.DAGImpl (DAGImpl.java:transition(1747)) - Vertex vertex_1463523512622_0001_1_00 [D65D4255FC664CD1A782B83C7638434D] completed., numCompletedVertices=1, numSuccessfulVertices=1, numFailedVertices=0, numKilledVertices=0, numVertices=2
2016-05-17 22:18:33,124 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event TASK_STARTED to history file
2016-05-17 22:18:33,124 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1463523512622_0001_1][Event:TASK_STARTED]: vertexName=DAF899C218864FA991BD8131DDD089DB, taskId=task_1463523512622_0001_1_01_000000, scheduledTime=1463523513124, launchTime=1463523513124
2016-05-17 22:18:33,125 INFO impl.TaskImpl (TaskImpl.java:handle(862)) - task_1463523512622_0001_1_01_000000 Task Transitioned from NEW to SCHEDULED due to event T_SCHEDULE
2016-05-17 22:18:33,125 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:stop(244)) - Ignoring stop request for containerId container_1463523512622_0001_00_000001. Relying on regular task shutdown for it to end
2016-05-17 22:18:33,125 INFO impl.DAGSchedulerNaturalOrderControlled (DAGSchedulerNaturalOrderControlled.java:scheduleTask(111)) - Scheduled vertex: vertex_1463523512622_0001_1_01 [DAF899C218864FA991BD8131DDD089DB]
2016-05-17 22:18:33,126 INFO container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1463523512622_0001_00_000001 transitioned from STOP_REQUESTED to STOPPING via event C_NM_STOP_SENT
2016-05-17 22:18:33,126 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:transition(1014)) - remoteTaskSpec:DAGName : [5F24C8AC118845E5B6CBE3B6796D5655/DBF10C5DC60F4509983187E52CD42569] test/(1/1) ...testSimpleBuffer2/simple2, VertexName: DAF899C218864FA991BD8131DDD089DB, VertexParallelism: 1, TaskAttemptID:attempt_1463523512622_0001_1_01_000000_0, processorName=cascading.flow.tez.FlowProcessor, inputSpecListSize=1, outputSpecListSize=1, inputSpecList=[{{ sourceVertexName=D65D4255FC664CD1A782B83C7638434D, physicalEdgeCount=1, inputClassName=org.apache.tez.runtime.library.input.OrderedGroupedKVInput }}, ], outputSpecList=[{{ destinationVertexName=08300649FE3340E28A95D849CC8CB7E3, physicalEdgeCount=0, outputClassName=org.apache.tez.mapreduce.output.MROutput }}, ]
2016-05-17 22:18:33,127 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:handle(692)) - attempt_1463523512622_0001_1_01_000000_0 TaskAttempt Transitioned from NEW to START_WAIT due to event TA_SCHEDULE
2016-05-17 22:18:33,127 INFO rm.TaskSchedulerEventHandler (TaskSchedulerEventHandler.java:handleEvent(149)) - Processing the event EventType: S_TA_LAUNCH_REQUEST
2016-05-17 22:18:33,128 INFO container.AMContainerImpl (AMContainerImpl.java:transition(521)) - Sending Launch Request for Container with id: container_1463523512622_0001_00_000002
2016-05-17 22:18:33,128 INFO container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1463523512622_0001_00_000002 transitioned from ALLOCATED to LAUNCHING via event C_LAUNCH_REQUEST
2016-05-17 22:18:33,130 INFO task.TezChild (TezChild.java:newTezChild(432)) - PID, containerIdentifier: null, container_1463523512622_0001_00_000002
2016-05-17 22:18:33,130 WARN impl.MetricsSystemImpl (MetricsSystemImpl.java:init(145)) - DAGAppMaster metrics system already initialized!
2016-05-17 22:18:33,131 INFO app.TaskAttemptListenerImpTezDag (TaskAttemptListenerImpTezDag.java:getTask(213)) - Container with id: container_1463523512622_0001_00_000001 is valid, but no longer registered, and will be killed
2016-05-17 22:18:33,132 INFO task.ContainerReporter (ContainerReporter.java:call(65)) - Got TaskUpdate for containerId= container_1463523512622_0001_00_000001: 12 ms after starting to poll. TaskInfo: shouldDie: true
2016-05-17 22:18:33,132 INFO task.TezChild (TezChild.java:run(207)) - ContainerTask returned shouldDie=true for container container_1463523512622_0001_00_000001, Exiting
2016-05-17 22:18:33,132 INFO task.TezChild (TezChild.java:shutdown(343)) - Shutdown invoked for container container_1463523512622_0001_00_000001
2016-05-17 22:18:33,132 INFO task.TezChild (TezChild.java:shutdown(345)) - Shutting down container container_1463523512622_0001_00_000001
2016-05-17 22:18:33,136 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event CONTAINER_LAUNCHED to history file
2016-05-17 22:18:33,136 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(270)) - ContainerExecutionResult for: container_1463523512622_0001_00_000001 = ContainerExecutionResult{exitStatus=SUCCESS, throwable=null, errorMessage='Asked to die by the AM'}
2016-05-17 22:18:33,136 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(274)) - Container: container_1463523512622_0001_00_000001 completed successfully
2016-05-17 22:18:33,136 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1463523512622_0001_1][Event:CONTAINER_LAUNCHED]: containerId=container_1463523512622_0001_00_000002, launchTime=1463523513136
2016-05-17 22:18:33,137 INFO container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1463523512622_0001_00_000002 transitioned from LAUNCHING to IDLE via event C_LAUNCHED
2016-05-17 22:18:33,137 INFO task.ContainerReporter (ContainerReporter.java:call(55)) - Attempting to fetch new task for container container_1463523512622_0001_00_000002
2016-05-17 22:18:33,137 INFO container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1463523512622_0001_00_000001 transitioned from STOPPING to COMPLETED via event C_COMPLETED
2016-05-17 22:18:33,138 INFO container.AMContainerImpl (AMContainerImpl.java:transition(827)) - Assigned taskAttempt + [attempt_1463523512622_0001_1_01_000000_0] to container: [container_1463523512622_0001_00_000002]
2016-05-17 22:18:33,138 INFO container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1463523512622_0001_00_000002 transitioned from IDLE to RUNNING via event C_PULL_TA
2016-05-17 22:18:33,139 INFO app.TaskAttemptListenerImpTezDag (TaskAttemptListenerImpTezDag.java:getTask(240)) - Container with id: container_1463523512622_0001_00_000002 given task: attempt_1463523512622_0001_1_01_000000_0
2016-05-17 22:18:33,139 INFO task.ContainerReporter (ContainerReporter.java:call(65)) - Got TaskUpdate for containerId= container_1463523512622_0001_00_000002: 0 ms after starting to poll. TaskInfo: shouldDie: false, currentTaskAttemptId: attempt_1463523512622_0001_1_01_000000_0
2016-05-17 22:18:33,139 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:transition(1146)) - TaskAttempt: [attempt_1463523512622_0001_1_01_000000_0] started. Is using containerId: [container_1463523512622_0001_00_000002] on NM: [127.0.0.1:0]
2016-05-17 22:18:33,140 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event TASK_ATTEMPT_STARTED to history file
2016-05-17 22:18:33 Starting to run new task attempt: attempt_1463523512622_0001_1_01_000000_0
2016-05-17 22:18:33,142 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(164)) - Redirecting log file based on addend: attempt_1463523512622_0001_1_01_000000_0
2016-05-17 22:18:33,143 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(179)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-17 22:18:33,143 INFO task.TezChild (TezChild.java:handleNewTaskCredentials(273)) - Refreshing UGI since Credentials have changed
2016-05-17 22:18:33,143 INFO task.TezChild (TezChild.java:handleNewTaskCredentials(276)) - Credentials : #Tokens=1, #SecretKeys=0
2016-05-17 22:18:33,143 INFO task.TezChild (TezChild.java:handleNewTaskLocalResources(302)) - Localizing additional local resources for Task : {}
2016-05-17 22:18:33,143 INFO task.TezChild (TezChild.java:handleNewTaskLocalResources(312)) - Done localizing additional resources
2016-05-17 22:18:33,143 INFO metrics.TaskCounterUpdater (TaskCounterUpdater.java:initResourceCalculatorPlugin(148)) - Using ResourceCalculatorProcessTree : [ ]
2016-05-17 22:18:33,143 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:<init>(149)) - Initializing LogicalIOProcessorRuntimeTask with TaskSpec: DAGName : [5F24C8AC118845E5B6CBE3B6796D5655/DBF10C5DC60F4509983187E52CD42569] test/(1/1) ...testSimpleBuffer2/simple2, VertexName: DAF899C218864FA991BD8131DDD089DB, VertexParallelism: 1, TaskAttemptID:attempt_1463523512622_0001_1_01_000000_0, processorName=cascading.flow.tez.FlowProcessor, inputSpecListSize=1, outputSpecListSize=1, inputSpecList=[{{ sourceVertexName=D65D4255FC664CD1A782B83C7638434D, physicalEdgeCount=1, inputClassName=org.apache.tez.runtime.library.input.OrderedGroupedKVInput }}, ], outputSpecList=[{{ destinationVertexName=08300649FE3340E28A95D849CC8CB7E3, physicalEdgeCount=0, outputClassName=org.apache.tez.mapreduce.output.MROutput }}, ]
2016-05-17 22:18:33,144 INFO resources.MemoryDistributor (MemoryDistributor.java:<init>(85)) - InitialMemoryDistributor (isEnabled=true) invoked with: numInputs=1, numOutputs=1, JVM.maxFree=704643072
2016-05-17 22:18:33,144 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1463523512622_0001_1][Event:TASK_ATTEMPT_STARTED]: vertexName=DAF899C218864FA991BD8131DDD089DB, taskAttemptId=attempt_1463523512622_0001_1_01_000000_0, startTime=1463523513139, containerId=container_1463523512622_0001_00_000002, nodeId=127.0.0.1:0, inProgressLogs=127.0.0.1:0/node/containerlogs/container_1463523512622_0001_00_000002/teamcity, completedLogs=
2016-05-17 22:18:33,144 INFO task.TezTaskRunner (TezTaskRunner.java:run(172)) - Initializing task, taskAttemptId=attempt_1463523512622_0001_1_01_000000_0
2016-05-17 22:18:33,144 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(191)) - Initializing LogicalProcessorIORuntimeTask
2016-05-17 22:18:33,145 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(195)) - Creating processor, processorClassName=cascading.flow.tez.FlowProcessor
2016-05-17 22:18:33,144 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:handle(692)) - attempt_1463523512622_0001_1_01_000000_0 TaskAttempt Transitioned from START_WAIT to RUNNING due to event TA_STARTED_REMOTELY
2016-05-17 22:18:33,145 INFO impl.TaskImpl (TaskImpl.java:handle(862)) - task_1463523512622_0001_1_01_000000 Task Transitioned from SCHEDULED to RUNNING due to event T_ATTEMPT_LAUNCHED
2016-05-17 22:18:33,147 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initializeLogicalIOProcessor(488)) - Initializing processor, processorClassName=cascading.flow.tez.FlowProcessor
2016-05-17 22:18:33,148 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(389)) - Initializing Input using InputSpec: { sourceVertexName=D65D4255FC664CD1A782B83C7638434D, physicalEdgeCount=1, inputClassName=org.apache.tez.runtime.library.input.OrderedGroupedKVInput }
2016-05-17 22:18:33,148 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:createInput(537)) - Creating Input
2016-05-17 22:18:33,148 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(397)) - Initializing Input with src edge: D65D4255FC664CD1A782B83C7638434D
2016-05-17 22:18:33,150 INFO orderedgrouped.MergeManager (MergeManager.java:getInitialMemoryRequirement(310)) - Initial Shuffle Memory Required: 634178752, based on INPUT_BUFFER_factor: 0.9
2016-05-17 22:18:33,150 INFO orderedgrouped.MergeManager (MergeManager.java:getInitialMemoryRequirement(320)) - Initial Memory required for final merged output: 0, using factor: 0.0
2016-05-17 22:18:33,152 INFO resources.MemoryDistributor$RequestorInfo (MemoryDistributor.java:<init>(233)) - Received request: 634178752, type: INPUT, componentVertexName: D65D4255FC664CD1A782B83C7638434D
2016-05-17 22:18:33,152 INFO tez.FlowProcessor (FlowProcessor.java:initialize(88)) - cascading version: 3.1.0
2016-05-17 22:18:33,152 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(403)) - Initialized Input with src edge: D65D4255FC664CD1A782B83C7638434D
2016-05-17 22:18:33,159 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(438)) - Initializing Output using OutputSpec: { destinationVertexName=08300649FE3340E28A95D849CC8CB7E3, physicalEdgeCount=0, outputClassName=org.apache.tez.mapreduce.output.MROutput }
2016-05-17 22:18:33,160 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:createOutput(565)) - Creating Output
2016-05-17 22:18:33,160 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(446)) - Initializing Output with dest edge: 08300649FE3340E28A95D849CC8CB7E3
2016-05-17 22:18:33,160 INFO output.MROutput (MROutput.java:initialize(347)) - Initializing Simple Output
2016-05-17 22:18:33,160 INFO resources.MemoryDistributor$RequestorInfo (MemoryDistributor.java:<init>(233)) - Received request: 0, type: OUTPUT, componentVertexName: 08300649FE3340E28A95D849CC8CB7E3
2016-05-17 22:18:33,166 INFO impl.TaskImpl (TaskImpl.java:getTaskAttemptTezEvents(481)) - TaskAttempt:attempt_1463523512622_0001_1_01_000000_0 sent events: (0-1)
2016-05-17 22:18:33,174 INFO tez.FlowProcessor (FlowProcessor.java:initialize(94)) - flow node id: DAF899C218864FA991BD8131DDD089DB, ordinal: 1
2016-05-17 22:18:33,174 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: DAF899C218864FA991BD8131DDD089DB, mem on start (mb), free: 389, total: 515, max: 672
2016-05-17 22:18:33,175 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initializeLogicalIOProcessor(491)) - Initialized processor, processorClassName=cascading.flow.tez.FlowProcessor
2016-05-17 22:18:33,176 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(220)) - Waiting for 2 initializers to finish
2016-05-17 22:18:33,176 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(220)) - Waiting for 1 initializers to finish
2016-05-17 22:18:33,185 INFO output.MROutput (MROutput.java:initialize(418)) - Initialized Simple Output, using_new_api: false
2016-05-17 22:18:33,185 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(452)) - Initialized Output with dest edge: 08300649FE3340E28A95D849CC8CB7E3
2016-05-17 22:18:33,185 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(233)) - All initializers finished
2016-05-17 22:18:33,185 INFO resources.MemoryDistributor (MemoryDistributor.java:makeInitialAllocations(124)) - Using Allocator class: org.apache.tez.runtime.library.resources.WeightedScalingMemoryDistributor
2016-05-17 22:18:33,186 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:initialProcessMemoryRequestContext(171)) - ScaleFactor: 12, for type: SORTED_MERGED_INPUT
2016-05-17 22:18:33,186 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:getRequestTypeForClass(197)) - Falling back to RequestType.OTHER for class: org.apache.tez.mapreduce.output.MROutput
2016-05-17 22:18:33,186 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:initialProcessMemoryRequestContext(171)) - ScaleFactor: 1, for type: OTHER
2016-05-17 22:18:33,186 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:computeReservedFraction(262)) - InitialReservationFraction=0.3, AdditionalReservationFractionForIOs=0.03, finalReserveFractionUsed=0.32999999999999996
2016-05-17 22:18:33,186 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:assignMemory(126)) - Scaling Requests. NumRequests: 2, numScaledRequests: 13, TotalRequested: 634178752, TotalRequestedScaled: 5.853957710769231E8, TotalJVMHeap: 704643072, TotalAvailable: 472110858, TotalRequested/TotalJVMHeap:0.90
2016-05-17 22:18:33,186 INFO resources.MemoryDistributor (MemoryDistributor.java:makeInitialAllocations(138)) - Informing: INPUT, D65D4255FC664CD1A782B83C7638434D, org.apache.tez.runtime.library.input.OrderedGroupedKVInput: requested=634178752, allocated=472110858
2016-05-17 22:18:33,186 INFO resources.MemoryDistributor (MemoryDistributor.java:makeInitialAllocations(138)) - Informing: OUTPUT, 08300649FE3340E28A95D849CC8CB7E3, org.apache.tez.mapreduce.output.MROutput: requested=0, allocated=0
2016-05-17 22:18:33,187 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(255)) - Starting Inputs/Outputs
2016-05-17 22:18:33,187 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(268)) - Input: D65D4255FC664CD1A782B83C7638434D being auto started by the framework. Subsequent instances will not be auto-started
2016-05-17 22:18:33,187 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(419)) - Starting Input with src edge: D65D4255FC664CD1A782B83C7638434D
2016-05-17 22:18:33,187 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(289)) - Num IOs determined for AutoStart: 1
2016-05-17 22:18:33,188 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(291)) - Waiting for 1 IOs to start
2016-05-17 22:18:33,189 INFO common.TezRuntimeUtils (TezRuntimeUtils.java:instantiateCombiner(65)) - No combiner specified via tez.runtime.combiner.class. Combiner will not be used
2016-05-17 22:18:33,189 INFO orderedgrouped.Shuffle (Shuffle.java:<init>(182)) - Shuffle assigned with 1 inputs, codec: NoneifileReadAhead: true
2016-05-17 22:18:33,189 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:<init>(158)) - ShuffleScheduler running for sourceVertex: D65D4255FC664CD1A782B83C7638434D with configuration: maxFetchFailuresBeforeReporting=5, reportReadErrorImmediately=true, maxFailedUniqueFetches=1, abortFailureLimit=30, maxMapRuntime=0, maxTaskOutputAtOnce=20
2016-05-17 22:18:33,190 INFO orderedgrouped.MergeManager (MergeManager.java:<init>(221)) - InitialRequest: ShuffleMem=634178752, postMergeMem=0, RuntimeTotalAvailable=472110858. Updated to: ShuffleMem=472110858, postMergeMem: 0
2016-05-17 22:18:33,190 INFO orderedgrouped.MergeManager (MergeManager.java:<init>(252)) - MergerManager: memoryLimit=472110858, maxSingleShuffleLimit=118027712, mergeThreshold=424899744, ioSortFactor=100, memToMemMergeOutputsThreshold=100
2016-05-17 22:18:33,190 INFO orderedgrouped.Shuffle (Shuffle.java:<init>(233)) - Num fetchers being started: 1
2016-05-17 22:18:33,195 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(421)) - Started Input with src edge: D65D4255FC664CD1A782B83C7638434D
2016-05-17 22:18:33,196 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(304)) - AutoStartComplete
2016-05-17 22:18:33,196 INFO task.TezTaskRunner (TezTaskRunner.java:run(175)) - Running task, taskAttemptId=attempt_1463523512622_0001_1_01_000000_0
2016-05-17 22:18:33,199 INFO graph.Hadoop2TezStreamGraph (Hadoop2TezStreamGraph.java:buildGraph(151)) - using streamed source: GroupBy(test)[by:[{1}:'num']]
2016-05-17 22:18:33,199 INFO orderedgrouped.ShuffleInputEventHandlerOrderedGrouped (ShuffleInputEventHandlerOrderedGrouped.java:processDataMovementEvent(84)) - DME srcIdx: 0, targetIdx: 0, attemptNum: 0, payload: [host: ip-10-123-148-208, port: 0, pathComponent: attempt_1463523512622_0001_1_00_000000_0_10031, runDuration: 203110, ]
2016-05-17 22:18:33,200 INFO tez.FlowProcessor (FlowProcessor.java:run(121)) - sourcing from: GroupBy(test)[by:[{1}:'num']] streamed: true, id: 165EBC44B5354F26A0228F229AE27CCF
2016-05-17 22:18:33,200 INFO tez.FlowProcessor (FlowProcessor.java:run(124)) - sinking to: Hfs["TextLine[['line']->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.5.x-tez-0.6.x/build/test/output/hadoop2-tez/bufferpipesplatform/testSimpleBuffer2/simple2"], id: 08300649FE3340E28A95D849CC8CB7E3
2016-05-17 22:18:33,201 INFO element.TezSinkStage (TezSinkStage.java:prepare(57)) - calling MROutput#start() on: Hfs["TextLine[['line']->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.5.x-tez-0.6.x/build/test/output/hadoop2-tez/bufferpipesplatform/testSimpleBuffer2/simple2"]
2016-05-17 22:18:33,201 INFO element.TezGroupGate (TezGroupGate.java:prepare(90)) - calling OrderedGroupedKVInput#start() on: GroupBy(test)[by:[{1}:'num']] 165EBC44B5354F26A0228F229AE27CCF, for 1 inputs
2016-05-17 22:18:33,204 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:logIndividualFetchComplete(250)) - Completed fetch for attempt: InputAttemptIdentifier [inputIdentifier=InputIdentifier [inputIndex=0], attemptNumber=0, pathComponent=attempt_1463523512622_0001_1_00_000000_0_10031] to DISK_DIRECT, CompressedSize=175, DecompressedSize=171,EndTime=1463523513204, TimeTaken=3, Rate=0.06 MB/s
2016-05-17 22:18:33,204 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:copySucceeded(213)) - All inputs fetched for input vertex : D65D4255FC664CD1A782B83C7638434D
2016-05-17 22:18:33,205 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:logProgress(263)) - copy(1 of 1. Transfer rate (CumulativeDataFetched/TimeSinceInputStarted)) 0.00 MB/s)
2016-05-17 22:18:33,206 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:freeHost(529)) - ip-10-123-148-208:0 freed by fetcher [D65D4255FC664CD1A782B83C7638434D] #6 in 5ms
2016-05-17 22:18:33,207 INFO orderedgrouped.Shuffle (Shuffle.java:cleanupFetchers(394)) - Shutdown..fetcher [D65D4255FC664CD1A782B83C7638434D] #6, status:false, isInterrupted:false
2016-05-17 22:18:33,207 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(854)) - finalMerge called with 0 in-memory map-outputs and 1 on-disk map-outputs
2016-05-17 22:18:33,208 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(951)) - Merging 1 files, 175 bytes from disk
2016-05-17 22:18:33,208 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(966)) - Merging 0 segments, 0 bytes from memory into reduce
2016-05-17 22:18:33,208 INFO impl.TezMerger (TezMerger.java:merge(560)) - Merging 1 sorted segments
2016-05-17 22:18:33,209 INFO impl.TezMerger (TezMerger.java:merge(665)) - Down to the last merge-pass, with 1 segments left of total size: 158 bytes
2016-05-17 22:18:33,209 INFO orderedgrouped.Shuffle (Shuffle.java:call(381)) - merge complete for input vertex : D65D4255FC664CD1A782B83C7638434D
2016-05-17 22:18:33,209 INFO orderedgrouped.Shuffle (Shuffle.java:onSuccess(488)) - Shuffle Runner thread complete
2016-05-17 22:18:33,210 INFO tez.FlowProcessor (FlowProcessor.java:waitForInputsReady(202)) - flow node id: DAF899C218864FA991BD8131DDD089DB, all 1 inputs ready in: 00:00:00.009
2016-05-17 22:18:33,212 INFO common.ValuesIterator (ValuesIterator.java:<init>(83)) - keyDeserializer=cascading.tuple.hadoop.io.KeyTupleDeserializer@6dbed0cc; valueDeserializer=cascading.tuple.hadoop.io.ValueTupleDeserializer@a7e395e
2016-05-17 22:18:33,215 INFO output.MROutput (MROutput.java:flush(534)) - Flushing Simple Output
2016-05-17 22:18:33,216 INFO output.MROutput (MROutput.java:flush(544)) - Flushed Simple Output
2016-05-17 22:18:33,217 INFO app.TaskAttemptListenerImpTezDag (TaskAttemptListenerImpTezDag.java:canCommit(353)) - Commit go/no-go request from attempt_1463523512622_0001_1_01_000000_0
2016-05-17 22:18:33,217 INFO impl.TaskImpl (TaskImpl.java:canCommit(750)) - attempt_1463523512622_0001_1_01_000000_0 given a go for committing the task output.
2016-05-17 22:18:33,218 INFO task.TezTaskRunner (TezTaskRunner.java:run(177)) - Closing task, taskAttemptId=attempt_1463523512622_0001_1_01_000000_0
2016-05-17 22:18:33,219 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: DAF899C218864FA991BD8131DDD089DB, mem on close (mb), free: 386, total: 515, max: 672
2016-05-17 22:18:33,219 INFO orderedgrouped.Shuffle (Shuffle.java:shutdown(320)) - Shutting down Shuffle for source: D65D4255FC664CD1A782B83C7638434D
2016-05-17 22:18:33,219 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(854)) - finalMerge called with 0 in-memory map-outputs and 0 on-disk map-outputs
2016-05-17 22:18:33,220 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(951)) - Merging 0 files, 0 bytes from disk
2016-05-17 22:18:33,220 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(966)) - Merging 0 segments, 0 bytes from memory into reduce
2016-05-17 22:18:33,221 INFO impl.TezMerger (TezMerger.java:merge(560)) - Merging 0 sorted segments
2016-05-17 22:18:33,221 INFO impl.TezMerger (TezMerger.java:merge(562)) - Nothing to merge. Returning an empty iterator
2016-05-17 22:18:33 Completed running task attempt: attempt_1463523512622_0001_1_01_000000_0
2016-05-17 22:18:33,222 INFO task.TezTaskRunner (TezTaskRunner.java:run(181)) - Task completed, taskAttemptId=attempt_1463523512622_0001_1_01_000000_0, fatalErrorOccurred=false
2016-05-17 22:18:33,224 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1463523512622_0001_1][Event:TASK_ATTEMPT_FINISHED]: vertexName=DAF899C218864FA991BD8131DDD089DB, taskAttemptId=attempt_1463523512622_0001_1_01_000000_0, startTime=1463523513139, finishTime=1463523513224, timeTaken=85, status=SUCCEEDED, errorEnum=, diagnostics=, counters=Counters: 46, File System Counters, FILE_BYTES_READ=207, FILE_BYTES_WRITTEN=210, FILE_READ_OPS=0, FILE_LARGE_READ_OPS=0, FILE_WRITE_OPS=0, org.apache.tez.common.counters.TaskCounter, REDUCE_INPUT_GROUPS=4, REDUCE_INPUT_RECORDS=13, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=13, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, GC_TIME_MILLIS=0, CPU_MILLISECONDS=0, PHYSICAL_MEMORY_BYTES=0, VIRTUAL_MEMORY_BYTES=0, COMMITTED_HEAP_BYTES=540540928, OUTPUT_RECORDS=18, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=175, SHUFFLE_BYTES=175, SHUFFLE_BYTES_DECOMPRESSED=171, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=175, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=17, MERGE_PHASE_TIME=20, FIRST_EVENT_RECEIVED=11, LAST_EVENT_RECEIVED=11, Shuffle Errors, BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0, cascading.flow.SliceCounters, Process_Begin_Time=1463523513210, Process_Duration=8, Process_End_Time=1463523513218, Read_Duration=1, Tuples_Read=13, Tuples_Written=18, Write_Duration=0, cascading.flow.StepCounters, Tuples_Written=18
2016-05-17 22:18:33,224 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:handle(692)) - attempt_1463523512622_0001_1_01_000000_0 TaskAttempt Transitioned from RUNNING to SUCCEEDED due to event TA_DONE
2016-05-17 22:18:33,225 INFO impl.TaskImpl (TaskImpl.java:transition(1052)) - Task succeeded with attempt attempt_1463523512622_0001_1_01_000000_0
2016-05-17 22:18:33,225 INFO rm.TaskSchedulerEventHandler (TaskSchedulerEventHandler.java:handleEvent(149)) - Processing the event EventType: S_TA_ENDED
2016-05-17 22:18:33,225 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event TASK_ATTEMPT_FINISHED to history file
2016-05-17 22:18:33,226 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1463523512622_0001_1][Event:TASK_FINISHED]: vertexName=DAF899C218864FA991BD8131DDD089DB, taskId=task_1463523512622_0001_1_01_000000, startTime=1463523513139, finishTime=1463523513225, timeTaken=86, status=SUCCEEDED, successfulAttemptID=attempt_1463523512622_0001_1_01_000000_0, diagnostics=, counters=Counters: 46, File System Counters, FILE_BYTES_READ=207, FILE_BYTES_WRITTEN=210, FILE_READ_OPS=0, FILE_LARGE_READ_OPS=0, FILE_WRITE_OPS=0, org.apache.tez.common.counters.TaskCounter, REDUCE_INPUT_GROUPS=4, REDUCE_INPUT_RECORDS=13, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=13, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, GC_TIME_MILLIS=0, CPU_MILLISECONDS=0, PHYSICAL_MEMORY_BYTES=0, VIRTUAL_MEMORY_BYTES=0, COMMITTED_HEAP_BYTES=540540928, OUTPUT_RECORDS=18, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=175, SHUFFLE_BYTES=175, SHUFFLE_BYTES_DECOMPRESSED=171, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=175, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=17, MERGE_PHASE_TIME=20, FIRST_EVENT_RECEIVED=11, LAST_EVENT_RECEIVED=11, Shuffle Errors, BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0, cascading.flow.SliceCounters, Process_Begin_Time=1463523513210, Process_Duration=8, Process_End_Time=1463523513218, Read_Duration=1, Tuples_Read=13, Tuples_Written=18, Write_Duration=0, cascading.flow.StepCounters, Tuples_Written=18
2016-05-17 22:18:33,226 INFO impl.TaskImpl (TaskImpl.java:handle(862)) - task_1463523512622_0001_1_01_000000 Task Transitioned from RUNNING to SUCCEEDED due to event T_ATTEMPT_SUCCEEDED
2016-05-17 22:18:33,226 INFO impl.VertexImpl (VertexImpl.java:transition(3536)) - Num completed Tasks for vertex_1463523512622_0001_1_01 [DAF899C218864FA991BD8131DDD089DB] : 1
2016-05-17 22:18:33,226 INFO impl.VertexImpl (VertexImpl.java:checkVertexForCompletion(1674)) - Vertex succeeded: vertex_1463523512622_0001_1_01 [DAF899C218864FA991BD8131DDD089DB]
2016-05-17 22:18:33,226 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event TASK_FINISHED to history file
2016-05-17 22:18:33,227 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event VERTEX_FINISHED to history file
2016-05-17 22:18:33,228 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1463523512622_0001_1][Event:VERTEX_FINISHED]: vertexName=DAF899C218864FA991BD8131DDD089DB, vertexId=vertex_1463523512622_0001_1_01, initRequestedTime=1463523512830, initedTime=1463523512846, startRequestedTime=1463523512864, startedTime=1463523512864, finishTime=1463523513226, timeTaken=362, status=SUCCEEDED, diagnostics=, counters=Counters: 46, File System Counters, FILE_BYTES_READ=207, FILE_BYTES_WRITTEN=210, FILE_READ_OPS=0, FILE_LARGE_READ_OPS=0, FILE_WRITE_OPS=0, org.apache.tez.common.counters.TaskCounter, REDUCE_INPUT_GROUPS=4, REDUCE_INPUT_RECORDS=13, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=13, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, GC_TIME_MILLIS=0, CPU_MILLISECONDS=0, PHYSICAL_MEMORY_BYTES=0, VIRTUAL_MEMORY_BYTES=0, COMMITTED_HEAP_BYTES=540540928, OUTPUT_RECORDS=18, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=175, SHUFFLE_BYTES=175, SHUFFLE_BYTES_DECOMPRESSED=171, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=175, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=17, MERGE_PHASE_TIME=20, FIRST_EVENT_RECEIVED=11, LAST_EVENT_RECEIVED=11, Shuffle Errors, BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0, cascading.flow.SliceCounters, Process_Begin_Time=1463523513210, Process_Duration=8, Process_End_Time=1463523513218, Read_Duration=1, Tuples_Read=13, Tuples_Written=18, Write_Duration=0, cascading.flow.StepCounters, Tuples_Written=18, vertexStats=firstTaskStartTime=1463523513139, firstTasksToStart=[ task_1463523512622_0001_1_01_000000 ], lastTaskFinishTime=1463523513224, lastTasksToFinish=[ task_1463523512622_0001_1_01_000000 ], minTaskDuration=85, maxTaskDuration=85, avgTaskDuration=85.0, numSuccessfulTasks=1, shortestDurationTasks=[ task_1463523512622_0001_1_01_000000 ], longestDurationTasks=[ task_1463523512622_0001_1_01_000000 ], vertexTaskStats={numFailedTasks=0, numSucceededTasks=1, numKilledTaskAttempts=0, numKilledTasks=0, numFailedTaskAttempts=0, numCompletedTasks=1}
2016-05-17 22:18:33,228 INFO impl.VertexImpl (VertexImpl.java:handle(1564)) - vertex_1463523512622_0001_1_01 [DAF899C218864FA991BD8131DDD089DB] transitioned from RUNNING to SUCCEEDED due to event V_TASK_COMPLETED
2016-05-17 22:18:33,230 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:cleanup(712)) - Final Counters : Counters: 46 [[File System Counters FILE_BYTES_READ=207, FILE_BYTES_WRITTEN=210, FILE_READ_OPS=0, FILE_LARGE_READ_OPS=0, FILE_WRITE_OPS=0][org.apache.tez.common.counters.TaskCounter REDUCE_INPUT_GROUPS=4, REDUCE_INPUT_RECORDS=13, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=13, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, GC_TIME_MILLIS=0, CPU_MILLISECONDS=0, PHYSICAL_MEMORY_BYTES=0, VIRTUAL_MEMORY_BYTES=0, COMMITTED_HEAP_BYTES=540540928, OUTPUT_RECORDS=18, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=175, SHUFFLE_BYTES=175, SHUFFLE_BYTES_DECOMPRESSED=171, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=175, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=17, MERGE_PHASE_TIME=20, FIRST_EVENT_RECEIVED=11, LAST_EVENT_RECEIVED=11][Shuffle Errors BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0][cascading.flow.SliceCounters Process_Begin_Time=1463523513210, Process_Duration=8, Process_End_Time=1463523513218, Read_Duration=1, Tuples_Read=13, Tuples_Written=18, Write_Duration=0][cascading.flow.StepCounters Tuples_Written=18]]
2016-05-17 22:18:33,230 INFO container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1463523512622_0001_00_000002 transitioned from RUNNING to IDLE via event C_TA_SUCCEEDED
2016-05-17 22:18:33,231 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1463523512622_0001_1][Event:CONTAINER_STOPPED]: containerId=container_1463523512622_0001_00_000002, stoppedTime=1463523513231, exitStatus=0
2016-05-17 22:18:33,231 INFO container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1463523512622_0001_00_000002 transitioned from IDLE to STOP_REQUESTED via event C_STOP_REQUEST
2016-05-17 22:18:33,232 INFO impl.DAGImpl (DAGImpl.java:transition(1747)) - Vertex vertex_1463523512622_0001_1_01 [DAF899C218864FA991BD8131DDD089DB] completed., numCompletedVertices=2, numSuccessfulVertices=2, numFailedVertices=0, numKilledVertices=0, numVertices=2
2016-05-17 22:18:33,232 INFO impl.DAGImpl (DAGImpl.java:commitOrAbortOutputs(831)) - Calling DAG commit/abort for dag: dag_1463523512622_0001_1
2016-05-17 22:18:33,231 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event CONTAINER_STOPPED to history file
2016-05-17 22:18:33,233 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1463523512622_0001_1][Event:DAG_COMMIT_STARTED]: dagID=dag_1463523512622_0001_1
2016-05-17 22:18:33,234 INFO impl.DAGImpl (DAGImpl.java:commitOrAbortOutputs(892)) - Committing output: 08300649FE3340E28A95D849CC8CB7E3 for vertex: vertex_1463523512622_0001_1_01 [DAF899C218864FA991BD8131DDD089DB]
2016-05-17 22:18:33,235 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(164)) - Redirecting log file based on addend:
2016-05-17 22:18:33,235 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(179)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-17 22:18:33,238 INFO task.ContainerReporter (ContainerReporter.java:call(55)) - Attempting to fetch new task for container container_1463523512622_0001_00_000002
2016-05-17 22:18:33,238 INFO app.TaskAttemptListenerImpTezDag (TaskAttemptListenerImpTezDag.java:getTask(213)) - Container with id: container_1463523512622_0001_00_000002 is valid, but no longer registered, and will be killed
2016-05-17 22:18:33,238 INFO task.ContainerReporter (ContainerReporter.java:call(65)) - Got TaskUpdate for containerId= container_1463523512622_0001_00_000002: 0 ms after starting to poll. TaskInfo: shouldDie: true
2016-05-17 22:18:33,238 INFO task.TezChild (TezChild.java:run(207)) - ContainerTask returned shouldDie=true for container container_1463523512622_0001_00_000002, Exiting
2016-05-17 22:18:33,238 INFO task.TezChild (TezChild.java:shutdown(343)) - Shutdown invoked for container container_1463523512622_0001_00_000002
2016-05-17 22:18:33,238 INFO task.TezChild (TezChild.java:shutdown(345)) - Shutting down container container_1463523512622_0001_00_000002
2016-05-17 22:18:33,243 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(270)) - ContainerExecutionResult for: container_1463523512622_0001_00_000002 = ContainerExecutionResult{exitStatus=SUCCESS, throwable=null, errorMessage='Asked to die by the AM'}
2016-05-17 22:18:33,243 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(274)) - Container: container_1463523512622_0001_00_000002 completed successfully
2016-05-17 22:18:33,253 INFO impl.DAGImpl (DAGImpl.java:commitOrAbortOutputs(871)) - No output committers for vertex: vertex_1463523512622_0001_1_00 [D65D4255FC664CD1A782B83C7638434D]
2016-05-17 22:18:33,256 INFO recovery.RecoveryService (RecoveryService.java:handle(306)) - DAG completed, dagId=dag_1463523512622_0001_1, queueSize=0
2016-05-17 22:18:33,257 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event DAG_FINISHED to history file
2016-05-17 22:18:33,258 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1463523512622_0001_1][Event:DAG_FINISHED]: dagId=dag_1463523512622_0001_1, startTime=1463523512827, finishTime=1463523513253, timeTaken=426, status=SUCCEEDED, diagnostics=, counters=Counters: 55, org.apache.tez.common.counters.DAGCounter, NUM_SUCCEEDED_TASKS=2, TOTAL_LAUNCHED_TASKS=2, RACK_LOCAL_TASKS=1, File System Counters, FILE_BYTES_READ=317, FILE_BYTES_WRITTEN=417, FILE_READ_OPS=0, FILE_LARGE_READ_OPS=0, FILE_WRITE_OPS=0, org.apache.tez.common.counters.TaskCounter, REDUCE_INPUT_GROUPS=4, REDUCE_INPUT_RECORDS=13, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=26, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, GC_TIME_MILLIS=146, CPU_MILLISECONDS=0, PHYSICAL_MEMORY_BYTES=0, VIRTUAL_MEMORY_BYTES=0, COMMITTED_HEAP_BYTES=1081081856, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=31, OUTPUT_BYTES=195, OUTPUT_BYTES_WITH_OVERHEAD=171, OUTPUT_BYTES_PHYSICAL=175, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=175, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_BYTES=175, SHUFFLE_BYTES_DECOMPRESSED=171, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=175, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=17, MERGE_PHASE_TIME=20, FIRST_EVENT_RECEIVED=11, LAST_EVENT_RECEIVED=11, Shuffle Errors, BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0, cascading.flow.SliceCounters, Process_Begin_Time=2927047026301, Process_Duration=12, Process_End_Time=2927047026313, Read_Duration=3, Tuples_Read=26, Tuples_Written=31, Write_Duration=0, cascading.flow.StepCounters, Tuples_Read=13, Tuples_Written=18
2016-05-17 22:18:33,259 INFO impl.DAGImpl (DAGImpl.java:finished(1180)) - DAG: dag_1463523512622_0001_1 finished with state: SUCCEEDED
2016-05-17 22:18:33,259 INFO impl.DAGImpl (DAGImpl.java:handle(973)) - dag_1463523512622_0001_1 transitioned from RUNNING to SUCCEEDED due to event DAG_VERTEX_COMPLETED
2016-05-17 22:18:33,259 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:stop(242)) - Ignoring stop request for containerId: container_1463523512622_0001_00_000002
2016-05-17 22:18:33,260 INFO container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1463523512622_0001_00_000002 transitioned from STOP_REQUESTED to COMPLETED via event C_COMPLETED
2016-05-17 22:18:33 Completed Dag: dag_1463523512622_0001_1
2016-05-17 22:18:33,262 INFO app.DAGAppMaster (DAGAppMaster.java:handle(579)) - DAG completed, dagId=dag_1463523512622_0001_1, dagState=SUCCEEDED
2016-05-17 22:18:33,262 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(164)) - Redirecting log file based on addend: dag_1463523512622_0001_1_post
2016-05-17 22:18:33,262 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(179)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-17 22:18:33,262 INFO app.DAGAppMaster (DAGAppMaster.java:handle(625)) - Waiting for next DAG to be submitted.
2016-05-17 22:18:33,269 INFO client.TezClient (TezClient.java:stop(495)) - Shutting down Tez Session, sessionName=(1/1) ...testSimpleBuffer2/simple2, applicationId=application_1463523512622_0001
2016-05-17 22:18:33,272 INFO client.DAGClientHandler (DAGClientHandler.java:shutdownAM(117)) - Received message to shutdown AM
2016-05-17 22:18:33,272 INFO rm.TaskSchedulerEventHandler (TaskSchedulerEventHandler.java:setShouldUnregisterFlag(580)) - TaskScheduler notified that it should unregister from RM
2016-05-17 22:18:33,272 INFO app.DAGAppMaster (DAGAppMaster.java:shutdownTezAM(1088)) - No current running DAG, shutting down the AM
2016-05-17 22:18:33,272 INFO app.DAGAppMaster (DAGAppMaster.java:shutdown(692)) - DAGAppMasterShutdownHandler invoked
2016-05-17 22:18:33,272 INFO app.DAGAppMaster (DAGAppMaster.java:shutdown(701)) - Handling DAGAppMaster shutdown
2016-05-17 22:18:33,274 INFO app.DAGAppMaster (DAGAppMaster.java:run(721)) - Sleeping for 5 seconds before shutting down
2016-05-17 22:18:33,274 INFO history.HistoryEventHandler (HistoryEventHandler.java:serviceStop(79)) - Stopping HistoryEventHandler
2016-05-17 22:18:33,275 INFO recovery.RecoveryService (RecoveryService.java:serviceStop(189)) - Stopping RecoveryService
2016-05-17 22:18:33,276 INFO recovery.RecoveryService (RecoveryService.java:serviceStop(192)) - Handle the remaining events in queue, queue size=0
2016-05-17 22:18:33,276 INFO recovery.RecoveryService (RecoveryService.java:run(166)) - EventQueue take interrupted. Returning
2016-05-17 22:18:33,277 INFO recovery.RecoveryService (RecoveryService.java:serviceStop(214)) - Closing Summary Stream
2016-05-17 22:18:33,277 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:serviceStop(118)) - Stopping SimpleHistoryLoggingService, eventQueueBacklog=0
2016-05-17 22:18:33,278 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:run(105)) - EventQueue take interrupted. Returning
2016-05-17 22:18:33,284 INFO app.DAGAppMaster (DAGAppMaster.java:run(1773)) - Completed deletion of tez scratch data dir, path=file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.5.x-tez-0.6.x/build/test/tmp/.staging/5F24C8AC118845E5B6CBE3B6796D5655/DBF10C5DC60F4509983187E52CD42569/.tez/application_1463523512622_0001
2016-05-17 22:18:33,285 INFO util.Hadoop18TapUtil (Hadoop18TapUtil.java:cleanTempPath(226)) - deleting temp path /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.5.x-tez-0.6.x/build/test/output/hadoop2-tez/bufferpipesplatform/testSimpleBuffer2/simple2/_temporary
2016-05-17 22:18:33,286 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [test] completed in: 00:00.682
2016-05-17 22:18:33,389 INFO cascading.PlatformTestCase (PlatformTestCase.java:tearDown(189)) - copying to local /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.5.x-tez-0.6.x/build/test/output//hadoop2-tez/bufferpipesplatform/testSimpleBuffer2/simple2
2016-05-17 22:18:33,415 INFO util.Util (Util.java:findMainClass(1374)) - using default application jar, may cause class not found exceptions on the cluster
2016-05-17 22:18:33,415 INFO planner.Hadoop2TezPlanner (Hadoop2TezPlanner.java:initialize(151)) - using application jar: /opt/concurrent/home/teamcity/.gradle/caches/modules-2/files-2.1/cascading/cascading-hadoop2-tez/3.1.0/226af02e24d5195c60e4ebe911d0c7c9780f59f0/cascading-hadoop2-tez-3.1.0.jar
2016-05-17 22:18:33,461 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [test] executed rule registry: NoHashJoinHadoop2TezRuleRegistry, completed as: SUCCESS, in: 00:00.020
2016-05-17 22:18:33,475 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [test] executed rule registry: HashJoinHadoop2TezRuleRegistry, completed as: SUCCESS, in: 00:00.034
2016-05-17 22:18:33,476 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [test] rule registry: NoHashJoinHadoop2TezRuleRegistry, supports assembly with steps: 1, nodes: 2
2016-05-17 22:18:33,476 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [test] rule registry: HashJoinHadoop2TezRuleRegistry, supports assembly with steps: 1, nodes: 2
2016-05-17 22:18:33,477 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [test] rule registry: NoHashJoinHadoop2TezRuleRegistry, result was selected using: 'default comparator: selects plan with fewest steps and fewest nodes'
2016-05-17 22:18:33,501 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [test] tap forcing step to tez local mode: /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.5.x-tez-0.6.x/build/classes/test/data/lower+upper.txt
2016-05-17 22:18:33,514 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [test] tap forcing step to tez local mode: /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.5.x-tez-0.6.x/build/test/output/hadoop2-tez/bufferpipesplatform/testSimpleBuffer3/simple3
2016-05-17 22:18:33,582 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [test] starting
2016-05-17 22:18:33,583 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [test] source: Hfs["TextLine[['offset', 'line' | long, String]->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.5.x-tez-0.6.x/build/classes/test/data/lower+upper.txt"]
2016-05-17 22:18:33,583 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [test] sink: Hfs["TextLine[['line']->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.5.x-tez-0.6.x/build/test/output/hadoop2-tez/bufferpipesplatform/testSimpleBuffer3/simple3"]
2016-05-17 22:18:33,583 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [test] parallel execution of steps is enabled: false
2016-05-17 22:18:33,584 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [test] executing total steps: 1
2016-05-17 22:18:33,584 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [test] allocating management threads: 1
2016-05-17 22:18:33,585 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [test] starting step: (1/1) ...testSimpleBuffer3/simple3
2016-05-17 22:18:33,585 WARN flow.Flow (BaseFlow.java:logWarn(1516)) - [test] 'yarn.timeline-service.enabled' is disabled, please enable to capture detailed metrics of completed flows, this may require starting the YARN timeline server daemon
2016-05-17 22:18:33,598 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [test] tez session mode enabled: true
2016-05-17 22:18:33,602 INFO client.TezClient (TezClient.java:<init>(153)) - Tez Client Version: [ component=tez-api, version=0.6.2, revision=c04283bf91fd96823eec37056d5c8b1a92728265, SCM-URL=scm:git:https://git-wip-us.apache.org/repos/asf/tez.git, buildTime=20150804-1436 ]
2016-05-17 22:18:33,603 INFO client.TezClient (TezClient.java:start(333)) - Session mode. Starting session.
2016-05-17 22:18:33,603 INFO client.TezClientUtils (TezClientUtils.java:setupTezJarsLocalResources(159)) - Ignoring 'tez.lib.uris' since 'tez.ignore.lib.uris' is set to true
2016-05-17 22:18:33,612 INFO client.TezClient (TezCommonUtils.java:createTezSystemStagingPath(118)) - Tez system stage directory file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.5.x-tez-0.6.x/build/test/tmp/.staging/86EF4021943745A2BAAB6F46343D991A/E27B02069D014C1E84A753853F2C4529/.tez/application_1463523513603_0001 doesn't exist and is created
2016-05-17 22:18:33,630 INFO client.LocalClient (LocalClient.java:createDAGAppMaster(328)) - DAGAppMaster thread has been created
2016-05-17 22:18:33,631 INFO client.LocalClient (LocalClient.java:startDAGAppMaster(251)) - DAGAppMaster is not created wait for 100ms...
2016-05-17 22:18:33,631 INFO client.LocalClient (LocalClient.java:run(275)) - Using staging directory: /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.5.x-tez-0.6.x/build/test/tmp/.staging/86EF4021943745A2BAAB6F46343D991A/E27B02069D014C1E84A753853F2C4529/.tez/application_1463523513603_0001
2016-05-17 22:18:33,636 INFO app.DAGAppMaster (DAGAppMaster.java:<init>(309)) - Created DAGAppMaster for application appattempt_1463523513603_0001_000000, versionInfo=[ component=tez-dag, version=0.6.2, revision=c04283bf91fd96823eec37056d5c8b1a92728265, SCM-URL=scm:git:https://git-wip-us.apache.org/repos/asf/tez.git, buildTime=20150804-1436 ]
2016-05-17 22:18:33,647 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(333)) - Comparing client version with AM version, clientVersion=0.6.2, AMVersion=0.6.2
2016-05-17 22:18:33,648 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(386)) - Adding session token to jobTokenSecretManager for application
2016-05-17 22:18:33,650 INFO node.AMNodeTracker (AMNodeTracker.java:serviceInit(77)) - blacklistDisablePercent is 33, blacklistingEnabled: false, maxTaskFailuresPerNode: 1
2016-05-17 22:18:33,650 INFO history.HistoryEventHandler (HistoryEventHandler.java:serviceInit(50)) - Initializing HistoryEventHandler
2016-05-17 22:18:33,650 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:serviceInit(66)) - Log file location for SimpleHistoryLoggingService not specified, defaulting to containerLogDir=/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.5.x-tez-0.6.x/build/test/tmp/.staging/86EF4021943745A2BAAB6F46343D991A/E27B02069D014C1E84A753853F2C4529/.tez/application_1463523513603_0001/localmode-log-dir
2016-05-17 22:18:33,651 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:serviceInit(88)) - Initializing SimpleHistoryLoggingService, logFileLocation=/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.5.x-tez-0.6.x/build/test/tmp/.staging/86EF4021943745A2BAAB6F46343D991A/E27B02069D014C1E84A753853F2C4529/.tez/application_1463523513603_0001/localmode-log-dir/history.txt.appattempt_1463523513603_0001_000000, maxErrors=10
2016-05-17 22:18:33,651 INFO recovery.RecoveryService (RecoveryService.java:serviceInit(109)) - Initializing RecoveryService
2016-05-17 22:18:33,651 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:N/A][Event:AM_LAUNCHED]: appAttemptId=appattempt_1463523513603_0001_000000, appSubmitTime=1463523513635, launchTime=1463523513635
2016-05-17 22:18:33,656 INFO history.HistoryEventHandler (HistoryEventHandler.java:serviceStart(73)) - Starting HistoryEventHandler
2016-05-17 22:18:33,659 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:serviceStart(95)) - Starting SimpleHistoryLoggingService
2016-05-17 22:18:33,662 INFO recovery.RecoveryService (RecoveryService.java:serviceStart(127)) - Starting RecoveryService
2016-05-17 22:18:33,665 INFO client.DAGClientServer (DAGClientServer.java:serviceStart(91)) - Instantiated DAGClientRPCServer at ip-10-123-148-208.ec2.internal/10.123.148.208:54172
2016-05-17 22:18:33,665 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event AM_LAUNCHED to history file
2016-05-17 22:18:33,666 WARN impl.MetricsConfig (MetricsConfig.java:loadFirst(124)) - Cannot locate configuration: tried hadoop-metrics2-dagappmaster.properties,hadoop-metrics2.properties
2016-05-17 22:18:33,669 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:N/A][Event:AM_STARTED]: appAttemptId=appattempt_1463523513603_0001_000000, startTime=1463523513669
2016-05-17 22:18:33,669 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event AM_STARTED to history file
2016-05-17 22:18:33,669 INFO app.DAGAppMaster (DAGAppMaster.java:serviceStart(1644)) - In Session mode. Waiting for DAG over RPC
2016-05-17 22:18:33,731 INFO client.LocalClient (LocalClient.java:startDAGAppMaster(236)) - DAGAppMaster state: IDLE
2016-05-17 22:18:33,731 INFO client.TezClient (TezClient.java:start(367)) - The url to track the Tez Session: N/A
2016-05-17 22:18:33,731 INFO client.TezClient (TezClient.java:submitDAGSession(405)) - Submitting dag to TezSession, sessionName=(1/1) ...testSimpleBuffer3/simple3, applicationId=application_1463523513603_0001, dagName=[86EF4021943745A2BAAB6F46343D991A/E27B02069D014C1E84A753853F2C4529] test/(1/1) ...testSimpleBuffer3/simple3
2016-05-17 22:18:33,749 INFO app.DAGAppMaster (DAGAppMaster.java:submitDAGToAppMaster(1126)) - Starting DAG submitted via RPC: [86EF4021943745A2BAAB6F46343D991A/E27B02069D014C1E84A753853F2C4529] test/(1/1) ...testSimpleBuffer3/simple3
2016-05-17 22:18:33,750 INFO app.DAGAppMaster (DAGAppMaster.java:generateDAGVizFile(870)) - Generating DAG graphviz file, dagId=dag_1463523513603_0001_1, filePath=/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.5.x-tez-0.6.x/build/test/tmp/.staging/86EF4021943745A2BAAB6F46343D991A/E27B02069D014C1E84A753853F2C4529/.tez/application_1463523513603_0001/localmode-log-dir/dag_1463523513603_0001_1.dot
2016-05-17 22:18:33,751 INFO app.DAGAppMaster (DAGAppMaster.java:writePBTextFile(887)) - Writing DAG plan to: /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.5.x-tez-0.6.x/build/test/tmp/.staging/86EF4021943745A2BAAB6F46343D991A/E27B02069D014C1E84A753853F2C4529/.tez/application_1463523513603_0001/localmode-log-dir/dag_1463523513603_0001_1-tez-dag.pb.txt
2016-05-17 22:18:33,761 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(164)) - Redirecting log file based on addend: dag_1463523513603_0001_1
2016-05-17 22:18:33,761 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(179)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-17 22:18:33,761 INFO app.DAGAppMaster (DAGAppMaster.java:startDAG(2051)) - Running DAG: [86EF4021943745A2BAAB6F46343D991A/E27B02069D014C1E84A753853F2C4529] test/(1/1) ...testSimpleBuffer3/simple3
2016-05-17 22:18:33 Running Dag: dag_1463523513603_0001_1
2016-05-17 22:18:33,775 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1463523513603_0001_1][Event:DAG_SUBMITTED]: dagID=dag_1463523513603_0001_1, submitTime=1463523513749
2016-05-17 22:18:33,775 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event DAG_SUBMITTED to history file
2016-05-17 22:18:33,775 INFO impl.VertexImpl (VertexImpl.java:setAdditionalOutputs(4033)) - setting additional outputs for vertex A95EBDDFBC0344098A63024BC7984A3D
2016-05-17 22:18:33,776 INFO impl.DAGImpl (DAGImpl.java:assignDAGScheduler(1381)) - Using DAG Scheduler: org.apache.tez.dag.app.dag.impl.DAGSchedulerNaturalOrderControlled
2016-05-17 22:18:33,776 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1463523513603_0001_1][Event:DAG_INITIALIZED]: dagID=dag_1463523513603_0001_1, initTime=1463523513775
2016-05-17 22:18:33,776 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event DAG_INITIALIZED to history file
2016-05-17 22:18:33,777 INFO impl.DAGImpl (DAGImpl.java:handle(973)) - dag_1463523513603_0001_1 transitioned from NEW to INITED due to event DAG_INIT
2016-05-17 22:18:33,778 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1463523513603_0001_1][Event:DAG_STARTED]: dagID=dag_1463523513603_0001_1, startTime=1463523513777
2016-05-17 22:18:33,778 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event DAG_STARTED to history file
2016-05-17 22:18:33,778 INFO impl.DAGImpl (DAGImpl.java:transition(1614)) - Added additional resources : [[]] to classpath
2016-05-17 22:18:33,778 INFO impl.DAGImpl (DAGImpl.java:handle(973)) - dag_1463523513603_0001_1 transitioned from INITED to RUNNING due to event DAG_START
2016-05-17 22:18:33,778 INFO impl.VertexImpl (VertexImpl.java:setupVertex(2068)) - Root Inputs exist for Vertex: AC44F8ABA3AC42CBB216039CD59B227D : {3C2EDA905A9E4693B68D30C5648DC8E5={InputName=3C2EDA905A9E4693B68D30C5648DC8E5}, {Descriptor=ClassName=org.apache.tez.mapreduce.input.MRInput, hasPayload=true}, {ControllerDescriptor=ClassName=org.apache.tez.mapreduce.common.MRInputAMSplitGenerator, hasPayload=false}}
2016-05-17 22:18:33,778 INFO impl.VertexImpl (VertexImpl.java:setupVertex(2078)) - Starting root input initializer for input: 3C2EDA905A9E4693B68D30C5648DC8E5, with class: [org.apache.tez.mapreduce.common.MRInputAMSplitGenerator]
2016-05-17 22:18:33,779 INFO client.TezClient (TezClient.java:submitDAGSession(478)) - Submitted dag to TezSession, sessionName=(1/1) ...testSimpleBuffer3/simple3, applicationId=application_1463523513603_0001, dagName=[86EF4021943745A2BAAB6F46343D991A/E27B02069D014C1E84A753853F2C4529] test/(1/1) ...testSimpleBuffer3/simple3
2016-05-17 22:18:33,779 INFO impl.VertexImpl (VertexImpl.java:assignVertexManager(2191)) - Setting vertexManager to RootInputVertexManager for vertex_1463523513603_0001_1_00 [AC44F8ABA3AC42CBB216039CD59B227D]
2016-05-17 22:18:33,779 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(2983)) - Num tasks is -1. Expecting VertexManager/InputInitializers/1-1 split to set #tasks for the vertex vertex_1463523513603_0001_1_00 [AC44F8ABA3AC42CBB216039CD59B227D]
2016-05-17 22:18:33,779 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(2987)) - Vertex will initialize from input initializer. vertex_1463523513603_0001_1_00 [AC44F8ABA3AC42CBB216039CD59B227D]
2016-05-17 22:18:33,779 INFO impl.VertexImpl (VertexImpl.java:setupInputInitializerManager(3933)) - Vertex will initialize via inputInitializers vertex_1463523513603_0001_1_00 [AC44F8ABA3AC42CBB216039CD59B227D]. Starting root input initializers: 1
2016-05-17 22:18:33,780 INFO impl.VertexImpl (VertexImpl.java:handle(1564)) - vertex_1463523513603_0001_1_00 [AC44F8ABA3AC42CBB216039CD59B227D] transitioned from NEW to INITIALIZING due to event V_INIT
2016-05-17 22:18:33,780 INFO impl.VertexImpl (VertexImpl.java:assignVertexManager(2203)) - Setting vertexManager to ShuffleVertexManager for vertex_1463523513603_0001_1_01 [A95EBDDFBC0344098A63024BC7984A3D]
2016-05-17 22:18:33,781 INFO dag.RootInputInitializerManager (RootInputInitializerManager.java:run(242)) - Starting InputInitializer for Input: 3C2EDA905A9E4693B68D30C5648DC8E5 on vertex vertex_1463523513603_0001_1_00 [AC44F8ABA3AC42CBB216039CD59B227D]
2016-05-17 22:18:33,783 INFO common.MRInputAMSplitGenerator (MRInputAMSplitGenerator.java:initialize(92)) - Emitting serialized splits: true
2016-05-17 22:18:33,783 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:initialize(696)) - Shuffle Vertex Manager: settings minFrac:0.25 maxFrac:0.75 auto:false desiredTaskIput:104857600 minTasks:1
2016-05-17 22:18:33,784 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(3014)) - Creating 1 tasks for vertex: vertex_1463523513603_0001_1_01 [A95EBDDFBC0344098A63024BC7984A3D]
2016-05-17 22:18:33,784 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(3026)) - Directly initializing vertex: vertex_1463523513603_0001_1_01 [A95EBDDFBC0344098A63024BC7984A3D]
2016-05-17 22:18:33,784 INFO impl.VertexImpl (VertexImpl.java:initializeCommitters(1915)) - Invoking committer inits for vertex, vertexId=vertex_1463523513603_0001_1_01 [A95EBDDFBC0344098A63024BC7984A3D]
2016-05-17 22:18:33,784 INFO impl.VertexImpl (VertexImpl.java:initializeCommitters(1927)) - Instantiating committer for output=B6674A59136C4FE092506740C4EA5E16, vertexId=vertex_1463523513603_0001_1_01 [A95EBDDFBC0344098A63024BC7984A3D], committerClass=org.apache.tez.mapreduce.committer.MROutputCommitter
2016-05-17 22:18:33,784 INFO common.MRInputAMSplitGenerator (MRInputAMSplitGenerator.java:initialize(110)) - Input 3C2EDA905A9E4693B68D30C5648DC8E5 asking for 1 tasks. Headroom: 672 Task Resource: 1024 waves: 1.7
2016-05-17 22:18:33,785 INFO impl.VertexImpl (VertexImpl.java:run(1945)) - Invoking committer init for output=B6674A59136C4FE092506740C4EA5E16, vertexId=vertex_1463523513603_0001_1_01 [A95EBDDFBC0344098A63024BC7984A3D]
2016-05-17 22:18:33,785 INFO common.MRInputAMSplitGenerator (MRInputAMSplitGenerator.java:initialize(121)) - Grouping input splits
2016-05-17 22:18:33,786 INFO hadoop.MRInputHelpers (MRInputHelpers.java:generateInputSplitsToMem(294)) - Generating mapred api input splits
2016-05-17 22:18:33,788 INFO committer.MROutputCommitter (MROutputCommitter.java:getOutputCommitter(145)) - OutputCommitter for outputName=B6674A59136C4FE092506740C4EA5E16, vertexName=A95EBDDFBC0344098A63024BC7984A3D, outputCommitterClass=org.apache.hadoop.mapred.FileOutputCommitter
2016-05-17 22:18:33,788 INFO hadoop.MRInputHelpers (MRInputHelpers.java:generateInputSplitsToMem(300)) - NumSplits: 1, SerializedSize: 412
2016-05-17 22:18:33,788 INFO impl.VertexImpl (VertexImpl.java:run(1949)) - Invoking committer setup for output=B6674A59136C4FE092506740C4EA5E16, vertexId=vertex_1463523513603_0001_1_01 [A95EBDDFBC0344098A63024BC7984A3D]
2016-05-17 22:18:33,790 INFO dag.RootInputInitializerManager (RootInputInitializerManager.java:onSuccess(272)) - Succeeded InputInitializer for Input: 3C2EDA905A9E4693B68D30C5648DC8E5 on vertex vertex_1463523513603_0001_1_00 [AC44F8ABA3AC42CBB216039CD59B227D]
2016-05-17 22:18:33,790 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event VERTEX_INITIALIZED to history file
2016-05-17 22:18:33,791 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1463523513603_0001_1][Event:VERTEX_INITIALIZED]: vertexName=A95EBDDFBC0344098A63024BC7984A3D, vertexId=vertex_1463523513603_0001_1_01, initRequestedTime=1463523513780, initedTime=1463523513790, numTasks=1, processorName=cascading.flow.tez.FlowProcessor, additionalInputsCount=0
2016-05-17 22:18:33,791 INFO impl.VertexImpl (VertexImpl.java:handle(1564)) - vertex_1463523513603_0001_1_01 [A95EBDDFBC0344098A63024BC7984A3D] transitioned from NEW to INITED due to event V_INIT
2016-05-17 22:18:33,791 INFO impl.VertexImpl (VertexImpl.java:canInitVertex(3195)) - Cannot init vertex: vertex_1463523513603_0001_1_00 [AC44F8ABA3AC42CBB216039CD59B227D] numTasks: -1 numUnitializedEdges: 0 numInitializedInputs: 0 initWaitsForRootInitializers: true
2016-05-17 22:18:33,792 INFO impl.VertexImpl (VertexImpl.java:setParallelism(1374)) - Got updated RootInputsSpecs: {3C2EDA905A9E4693B68D30C5648DC8E5=forAllWorkUnits=true, update=[1]}
2016-05-17 22:18:33,792 INFO impl.VertexImpl (VertexImpl.java:setParallelism(1393)) - Vertex vertex_1463523513603_0001_1_00 [AC44F8ABA3AC42CBB216039CD59B227D] parallelism set to 1
2016-05-17 22:18:33,792 INFO impl.VertexImpl (VertexImpl.java:canInitVertex(3195)) - Cannot init vertex: vertex_1463523513603_0001_1_00 [AC44F8ABA3AC42CBB216039CD59B227D] numTasks: 1 numUnitializedEdges: 0 numInitializedInputs: 0 initWaitsForRootInitializers: true
2016-05-17 22:18:33,792 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1463523513603_0001_1][Event:VERTEX_DATA_MOVEMENT_EVENTS_GENERATED]: vertexId=vertex_1463523513603_0001_1_00, eventCount=1
2016-05-17 22:18:33,795 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1463523513603_0001_1][Event:VERTEX_INITIALIZED]: vertexName=AC44F8ABA3AC42CBB216039CD59B227D, vertexId=vertex_1463523513603_0001_1_00, initRequestedTime=1463523513778, initedTime=1463523513794, numTasks=1, processorName=cascading.flow.tez.FlowProcessor, additionalInputsCount=1
2016-05-17 22:18:33,797 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event VERTEX_INITIALIZED to history file
2016-05-17 22:18:33,798 INFO impl.VertexImpl (VertexImpl.java:startIfPossible(3061)) - Triggering start event for vertex: vertex_1463523513603_0001_1_00 [AC44F8ABA3AC42CBB216039CD59B227D] with distanceFromRoot: 0
2016-05-17 22:18:33,799 INFO impl.VertexImpl (VertexImpl.java:handle(1564)) - vertex_1463523513603_0001_1_00 [AC44F8ABA3AC42CBB216039CD59B227D] transitioned from INITIALIZING to INITED due to event V_ROOT_INPUT_INITIALIZED
2016-05-17 22:18:33,799 INFO impl.ImmediateStartVertexManager (ImmediateStartVertexManager.java:scheduleTasks(99)) - Starting 1 in AC44F8ABA3AC42CBB216039CD59B227D
2016-05-17 22:18:33,799 INFO impl.VertexImpl (VertexImpl.java:scheduleTasks(1250)) - Routing pending task events for vertex: vertex_1463523513603_0001_1_00 [AC44F8ABA3AC42CBB216039CD59B227D]
2016-05-17 22:18:33,800 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event VERTEX_STARTED to history file
2016-05-17 22:18:33,800 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1463523513603_0001_1][Event:VERTEX_STARTED]: vertexId=vertex_1463523513603_0001_1_00, startRequestedTime=1463523513799, startedTime=1463523513799
2016-05-17 22:18:33,801 INFO impl.VertexImpl (VertexImpl.java:handle(1564)) - vertex_1463523513603_0001_1_00 [AC44F8ABA3AC42CBB216039CD59B227D] transitioned from INITED to RUNNING due to event V_START
2016-05-17 22:18:33,801 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event TASK_STARTED to history file
2016-05-17 22:18:33,802 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1463523513603_0001_1][Event:TASK_STARTED]: vertexName=AC44F8ABA3AC42CBB216039CD59B227D, taskId=task_1463523513603_0001_1_00_000000, scheduledTime=1463523513801, launchTime=1463523513801
2016-05-17 22:18:33,802 INFO impl.TaskImpl (TaskImpl.java:handle(862)) - task_1463523513603_0001_1_00_000000 Task Transitioned from NEW to SCHEDULED due to event T_SCHEDULE
2016-05-17 22:18:33,802 INFO impl.VertexImpl (VertexImpl.java:transition(3155)) - Source vertex started: vertex_1463523513603_0001_1_00 for vertex: vertex_1463523513603_0001_1_01 [A95EBDDFBC0344098A63024BC7984A3D] numStartedSources: 1 numSources: 1
2016-05-17 22:18:33,802 INFO impl.VertexImpl (VertexImpl.java:startIfPossible(3061)) - Triggering start event for vertex: vertex_1463523513603_0001_1_01 [A95EBDDFBC0344098A63024BC7984A3D] with distanceFromRoot: 1
2016-05-17 22:18:33,803 INFO impl.DAGSchedulerNaturalOrderControlled (DAGSchedulerNaturalOrderControlled.java:scheduleTask(111)) - Scheduled vertex: vertex_1463523513603_0001_1_00 [AC44F8ABA3AC42CBB216039CD59B227D]
2016-05-17 22:18:33,803 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:handleVertexStateUpdate(720)) - Received configured notification : CONFIGURED for vertex: AC44F8ABA3AC42CBB216039CD59B227D in vertex: A95EBDDFBC0344098A63024BC7984A3D
2016-05-17 22:18:33,804 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:onVertexStarted(355)) - OnVertexStarted vertex: A95EBDDFBC0344098A63024BC7984A3D with 1 source tasks and 1 pending tasks
2016-05-17 22:18:33,804 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event VERTEX_STARTED to history file
2016-05-17 22:18:33,804 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1463523513603_0001_1][Event:VERTEX_STARTED]: vertexId=vertex_1463523513603_0001_1_01, startRequestedTime=1463523513803, startedTime=1463523513803
2016-05-17 22:18:33,805 INFO impl.VertexImpl (VertexImpl.java:handle(1564)) - vertex_1463523513603_0001_1_01 [A95EBDDFBC0344098A63024BC7984A3D] transitioned from INITED to RUNNING due to event V_START
2016-05-17 22:18:33,805 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:transition(1014)) - remoteTaskSpec:DAGName : [86EF4021943745A2BAAB6F46343D991A/E27B02069D014C1E84A753853F2C4529] test/(1/1) ...testSimpleBuffer3/simple3, VertexName: AC44F8ABA3AC42CBB216039CD59B227D, VertexParallelism: 1, TaskAttemptID:attempt_1463523513603_0001_1_00_000000_0, processorName=cascading.flow.tez.FlowProcessor, inputSpecListSize=1, outputSpecListSize=1, inputSpecList=[{{ sourceVertexName=3C2EDA905A9E4693B68D30C5648DC8E5, physicalEdgeCount=1, inputClassName=org.apache.tez.mapreduce.input.MRInput }}, ], outputSpecList=[{{ destinationVertexName=A95EBDDFBC0344098A63024BC7984A3D, physicalEdgeCount=1, outputClassName=org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput }}, ]
2016-05-17 22:18:33,806 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:handle(692)) - attempt_1463523513603_0001_1_00_000000_0 TaskAttempt Transitioned from NEW to START_WAIT due to event TA_SCHEDULE
2016-05-17 22:18:33,806 INFO rm.TaskSchedulerEventHandler (TaskSchedulerEventHandler.java:handleEvent(149)) - Processing the event EventType: S_TA_LAUNCH_REQUEST
2016-05-17 22:18:33,807 INFO node.AMNodeTracker (AMNodeTracker.java:nodeSeen(91)) - Adding new node: 127.0.0.1:0
2016-05-17 22:18:33,807 INFO container.AMContainerHelpers (AMContainerHelpers.java:createCommonContainerLaunchContext(112)) - Adding #1 tokens and #0 secret keys for NM use for launching container
2016-05-17 22:18:33,807 INFO container.AMContainerHelpers (AMContainerHelpers.java:createCommonContainerLaunchContext(122)) - Putting shuffle token in serviceData
2016-05-17 22:18:33,808 INFO container.AMContainerImpl (AMContainerImpl.java:transition(521)) - Sending Launch Request for Container with id: container_1463523513603_0001_00_000001
2016-05-17 22:18:33,808 INFO container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1463523513603_0001_00_000001 transitioned from ALLOCATED to LAUNCHING via event C_LAUNCH_REQUEST
2016-05-17 22:18:33,810 INFO task.TezChild (TezChild.java:newTezChild(432)) - PID, containerIdentifier: null, container_1463523513603_0001_00_000001
2016-05-17 22:18:33,810 WARN impl.MetricsSystemImpl (MetricsSystemImpl.java:init(145)) - DAGAppMaster metrics system already initialized!
2016-05-17 22:18:33,811 INFO container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1463523513603_0001_00_000001 transitioned from LAUNCHING to IDLE via event C_LAUNCHED
2016-05-17 22:18:33,811 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event CONTAINER_LAUNCHED to history file
2016-05-17 22:18:33,811 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1463523513603_0001_1][Event:CONTAINER_LAUNCHED]: containerId=container_1463523513603_0001_00_000001, launchTime=1463523513810
2016-05-17 22:18:33,812 INFO task.ContainerReporter (ContainerReporter.java:call(55)) - Attempting to fetch new task for container container_1463523513603_0001_00_000001
2016-05-17 22:18:33,814 INFO container.AMContainerImpl (AMContainerImpl.java:transition(827)) - Assigned taskAttempt + [attempt_1463523513603_0001_1_00_000000_0] to container: [container_1463523513603_0001_00_000001]
2016-05-17 22:18:33,814 INFO container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1463523513603_0001_00_000001 transitioned from IDLE to RUNNING via event C_PULL_TA
2016-05-17 22:18:33,816 INFO app.TaskAttemptListenerImpTezDag (TaskAttemptListenerImpTezDag.java:getTask(240)) - Container with id: container_1463523513603_0001_00_000001 given task: attempt_1463523513603_0001_1_00_000000_0
2016-05-17 22:18:33,817 INFO task.ContainerReporter (ContainerReporter.java:call(65)) - Got TaskUpdate for containerId= container_1463523513603_0001_00_000001: 0 ms after starting to poll. TaskInfo: shouldDie: false, currentTaskAttemptId: attempt_1463523513603_0001_1_00_000000_0
2016-05-17 22:18:33,817 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:transition(1146)) - TaskAttempt: [attempt_1463523513603_0001_1_00_000000_0] started. Is using containerId: [container_1463523513603_0001_00_000001] on NM: [127.0.0.1:0]
2016-05-17 22:18:33,817 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1463523513603_0001_1][Event:TASK_ATTEMPT_STARTED]: vertexName=AC44F8ABA3AC42CBB216039CD59B227D, taskAttemptId=attempt_1463523513603_0001_1_00_000000_0, startTime=1463523513816, containerId=container_1463523513603_0001_00_000001, nodeId=127.0.0.1:0, inProgressLogs=127.0.0.1:0/node/containerlogs/container_1463523513603_0001_00_000001/teamcity, completedLogs=
2016-05-17 22:18:33 Starting to run new task attempt: attempt_1463523513603_0001_1_00_000000_0
2016-05-17 22:18:33,817 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:handle(692)) - attempt_1463523513603_0001_1_00_000000_0 TaskAttempt Transitioned from START_WAIT to RUNNING due to event TA_STARTED_REMOTELY
2016-05-17 22:18:33,817 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event TASK_ATTEMPT_STARTED to history file
2016-05-17 22:18:33,817 INFO impl.TaskImpl (TaskImpl.java:handle(862)) - task_1463523513603_0001_1_00_000000 Task Transitioned from SCHEDULED to RUNNING due to event T_ATTEMPT_LAUNCHED
2016-05-17 22:18:33,817 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(164)) - Redirecting log file based on addend: attempt_1463523513603_0001_1_00_000000_0
2016-05-17 22:18:33,818 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(179)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-17 22:18:33,818 INFO task.TezChild (TezChild.java:handleNewTaskCredentials(273)) - Refreshing UGI since Credentials have changed
2016-05-17 22:18:33,818 INFO task.TezChild (TezChild.java:handleNewTaskCredentials(276)) - Credentials : #Tokens=1, #SecretKeys=0
2016-05-17 22:18:33,818 INFO task.TezChild (TezChild.java:handleNewTaskLocalResources(302)) - Localizing additional local resources for Task : {}
2016-05-17 22:18:33,818 INFO task.TezChild (TezChild.java:handleNewTaskLocalResources(312)) - Done localizing additional resources
2016-05-17 22:18:33,818 INFO metrics.TaskCounterUpdater (TaskCounterUpdater.java:initResourceCalculatorPlugin(148)) - Using ResourceCalculatorProcessTree : [ ]
2016-05-17 22:18:33,818 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:<init>(149)) - Initializing LogicalIOProcessorRuntimeTask with TaskSpec: DAGName : [86EF4021943745A2BAAB6F46343D991A/E27B02069D014C1E84A753853F2C4529] test/(1/1) ...testSimpleBuffer3/simple3, VertexName: AC44F8ABA3AC42CBB216039CD59B227D, VertexParallelism: 1, TaskAttemptID:attempt_1463523513603_0001_1_00_000000_0, processorName=cascading.flow.tez.FlowProcessor, inputSpecListSize=1, outputSpecListSize=1, inputSpecList=[{{ sourceVertexName=3C2EDA905A9E4693B68D30C5648DC8E5, physicalEdgeCount=1, inputClassName=org.apache.tez.mapreduce.input.MRInput }}, ], outputSpecList=[{{ destinationVertexName=A95EBDDFBC0344098A63024BC7984A3D, physicalEdgeCount=1, outputClassName=org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput }}, ]
2016-05-17 22:18:33,819 INFO resources.MemoryDistributor (MemoryDistributor.java:<init>(85)) - InitialMemoryDistributor (isEnabled=true) invoked with: numInputs=1, numOutputs=1, JVM.maxFree=704643072
2016-05-17 22:18:33,819 INFO task.TezTaskRunner (TezTaskRunner.java:run(172)) - Initializing task, taskAttemptId=attempt_1463523513603_0001_1_00_000000_0
2016-05-17 22:18:33,819 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(191)) - Initializing LogicalProcessorIORuntimeTask
2016-05-17 22:18:33,821 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(195)) - Creating processor, processorClassName=cascading.flow.tez.FlowProcessor
2016-05-17 22:18:33,822 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initializeLogicalIOProcessor(488)) - Initializing processor, processorClassName=cascading.flow.tez.FlowProcessor
2016-05-17 22:18:33,822 INFO impl.TaskImpl (TaskImpl.java:getTaskAttemptTezEvents(481)) - TaskAttempt:attempt_1463523513603_0001_1_00_000000_0 sent events: (0-1)
2016-05-17 22:18:33,823 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(438)) - Initializing Output using OutputSpec: { destinationVertexName=A95EBDDFBC0344098A63024BC7984A3D, physicalEdgeCount=1, outputClassName=org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput }
2016-05-17 22:18:33,824 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [test] submitted tez dag to app master: application_1463523513603_0001, with dag id: dag_1463523513603_0001_1
2016-05-17 22:18:33,828 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(389)) - Initializing Input using InputSpec: { sourceVertexName=3C2EDA905A9E4693B68D30C5648DC8E5, physicalEdgeCount=1, inputClassName=org.apache.tez.mapreduce.input.MRInput }
2016-05-17 22:18:33,829 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:createInput(537)) - Creating Input
2016-05-17 22:18:33,829 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(397)) - Initializing Input with src edge: 3C2EDA905A9E4693B68D30C5648DC8E5
2016-05-17 22:18:33,828 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:createOutput(565)) - Creating Output
2016-05-17 22:18:33,831 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(446)) - Initializing Output with dest edge: A95EBDDFBC0344098A63024BC7984A3D
2016-05-17 22:18:33,830 INFO resources.MemoryDistributor$RequestorInfo (MemoryDistributor.java:<init>(233)) - Received request: 0, type: INPUT, componentVertexName: 3C2EDA905A9E4693B68D30C5648DC8E5
2016-05-17 22:18:33,829 INFO tez.FlowProcessor (FlowProcessor.java:initialize(88)) - cascading version: 3.1.0
2016-05-17 22:18:33,833 INFO impl.ExternalSorter (ExternalSorter.java:getInitialMemoryRequirement(293)) - Requested SortBufferSize (tez.runtime.io.sort.mb): 100
2016-05-17 22:18:33,833 INFO resources.MemoryDistributor$RequestorInfo (MemoryDistributor.java:<init>(233)) - Received request: 104857600, type: OUTPUT, componentVertexName: A95EBDDFBC0344098A63024BC7984A3D
2016-05-17 22:18:33,839 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(452)) - Initialized Output with dest edge: A95EBDDFBC0344098A63024BC7984A3D
2016-05-17 22:18:33,841 INFO input.MRInput (MRInput.java:initialize(440)) - Using New mapreduce API: false, split information via event: true
2016-05-17 22:18:33,841 INFO input.MRInput (MRInput.java:initializeInternal(493)) - Initialzed MRInput: 3C2EDA905A9E4693B68D30C5648DC8E5
2016-05-17 22:18:33,842 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(403)) - Initialized Input with src edge: 3C2EDA905A9E4693B68D30C5648DC8E5
2016-05-17 22:18:33,842 INFO tez.FlowProcessor (FlowProcessor.java:initialize(94)) - flow node id: AC44F8ABA3AC42CBB216039CD59B227D, ordinal: 0
2016-05-17 22:18:33,842 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: AC44F8ABA3AC42CBB216039CD59B227D, mem on start (mb), free: 293, total: 515, max: 672
2016-05-17 22:18:33,843 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initializeLogicalIOProcessor(491)) - Initialized processor, processorClassName=cascading.flow.tez.FlowProcessor
2016-05-17 22:18:33,843 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(220)) - Waiting for 2 initializers to finish
2016-05-17 22:18:33,843 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(220)) - Waiting for 1 initializers to finish
2016-05-17 22:18:33,843 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(233)) - All initializers finished
2016-05-17 22:18:33,843 INFO resources.MemoryDistributor (MemoryDistributor.java:makeInitialAllocations(124)) - Using Allocator class: org.apache.tez.runtime.library.resources.WeightedScalingMemoryDistributor
2016-05-17 22:18:33,843 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:getRequestTypeForClass(197)) - Falling back to RequestType.OTHER for class: org.apache.tez.mapreduce.input.MRInput
2016-05-17 22:18:33,843 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:initialProcessMemoryRequestContext(171)) - ScaleFactor: 1, for type: OTHER
2016-05-17 22:18:33,843 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:initialProcessMemoryRequestContext(171)) - ScaleFactor: 12, for type: SORTED_OUTPUT
2016-05-17 22:18:33,843 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:computeReservedFraction(262)) - InitialReservationFraction=0.3, AdditionalReservationFractionForIOs=0.03, finalReserveFractionUsed=0.32999999999999996
2016-05-17 22:18:33,844 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:assignMemory(126)) - Scaling Requests. NumRequests: 2, numScaledRequests: 13, TotalRequested: 104857600, TotalRequestedScaled: 9.679163076923077E7, TotalJVMHeap: 704643072, TotalAvailable: 472110858, TotalRequested/TotalJVMHeap:0.15
2016-05-17 22:18:33,844 INFO resources.MemoryDistributor (MemoryDistributor.java:makeInitialAllocations(138)) - Informing: INPUT, 3C2EDA905A9E4693B68D30C5648DC8E5, org.apache.tez.mapreduce.input.MRInput: requested=0, allocated=0
2016-05-17 22:18:33,844 INFO resources.MemoryDistributor (MemoryDistributor.java:makeInitialAllocations(138)) - Informing: OUTPUT, A95EBDDFBC0344098A63024BC7984A3D, org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput: requested=104857600, allocated=104857600
2016-05-17 22:18:33,844 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(255)) - Starting Inputs/Outputs
2016-05-17 22:18:33,844 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(268)) - Input: 3C2EDA905A9E4693B68D30C5648DC8E5 being auto started by the framework. Subsequent instances will not be auto-started
2016-05-17 22:18:33,844 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(419)) - Starting Input with src edge: 3C2EDA905A9E4693B68D30C5648DC8E5
2016-05-17 22:18:33,845 INFO input.MRInput (MRInput.java:start(450)) - MRInput setup to received events1
2016-05-17 22:18:33,845 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(421)) - Started Input with src edge: 3C2EDA905A9E4693B68D30C5648DC8E5
2016-05-17 22:18:33,845 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(289)) - Num IOs determined for AutoStart: 1
2016-05-17 22:18:33,845 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(291)) - Waiting for 1 IOs to start
2016-05-17 22:18:33,845 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(304)) - AutoStartComplete
2016-05-17 22:18:33,846 INFO task.TezTaskRunner (TezTaskRunner.java:run(175)) - Running task, taskAttemptId=attempt_1463523513603_0001_1_00_000000_0
2016-05-17 22:18:33,846 INFO input.MRInput (MRInput.java:initFromEventInternal(619)) - Initializing RecordReader from event
2016-05-17 22:18:33,847 INFO input.MRInput (MRInput.java:initFromEventInternal(630)) - Split Details -> SplitClass: org.apache.hadoop.mapred.split.TezGroupedSplit, OldSplit: org.apache.hadoop.mapred.split.TezGroupedSplit@2d977ced
2016-05-17 22:18:33,848 INFO lib.MRReaderMapred (MRReaderMapred.java:setIncrementalConfigParams(163)) - Processing split: org.apache.hadoop.mapred.split.TezGroupedSplit@2d977ced
2016-05-17 22:18:33,848 INFO input.MRInput (MRInput.java:initFromEventInternal(634)) - Initialized RecordReader from event
2016-05-17 22:18:33,848 INFO input.MRInput (MRInput.java:processSplitEvent(586)) - Notifying on RecordReader Initialized
2016-05-17 22:18:33,849 INFO graph.Hadoop2TezStreamGraph (Hadoop2TezStreamGraph.java:buildGraph(151)) - using streamed source: Hfs["TextLine[['offset', 'line' | long, String]->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.5.x-tez-0.6.x/build/classes/test/data/lower+upper.txt"]
2016-05-17 22:18:33,850 INFO tez.FlowProcessor (FlowProcessor.java:run(121)) - sourcing from: Hfs["TextLine[['offset', 'line' | long, String]->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.5.x-tez-0.6.x/build/classes/test/data/lower+upper.txt"] streamed: true, id: 3C2EDA905A9E4693B68D30C5648DC8E5
2016-05-17 22:18:33,850 INFO tez.FlowProcessor (FlowProcessor.java:run(124)) - sinking to: GroupBy(test)[by:[{1}:'num']], id: 5DBC737D09BF4B5EB74DE9B1EF8311A7
2016-05-17 22:18:33,851 INFO element.TezGroupGate (TezGroupGate.java:prepare(98)) - calling OrderedPartitionedKVOutput#start() on: GroupBy(test)[by:[{1}:'num']] 5DBC737D09BF4B5EB74DE9B1EF8311A7
2016-05-17 22:18:33,852 INFO impl.ExternalSorter (ExternalSorter.java:<init>(167)) - keySerializer=cascading.tuple.hadoop.io.KeyTupleSerializer@ca6a4aa; valueSerializer=cascading.tuple.hadoop.io.ValueTupleSerializer@10849b63; comparator=cascading.tuple.hadoop.util.TupleComparator@71f5890d; conf=cascading.tuple.hadoop.TupleSerialization,org.apache.hadoop.io.serializer.WritableSerialization,org.apache.hadoop.io.serializer.avro.AvroSpecificSerialization,org.apache.hadoop.io.serializer.avro.AvroReflectSerialization
2016-05-17 22:18:33,852 INFO impl.ExternalSorter (ExternalSorter.java:<init>(224)) - Instantiating Partitioner: [cascading.tuple.tez.util.TuplePartitioner]
2016-05-17 22:18:33,853 INFO common.TezRuntimeUtils (TezRuntimeUtils.java:instantiatePartitioner(109)) - Using partitioner class: cascading.tuple.tez.util.TuplePartitioner
2016-05-17 22:18:33,853 INFO common.TezRuntimeUtils (TezRuntimeUtils.java:instantiateCombiner(65)) - No combiner specified via tez.runtime.combiner.class. Combiner will not be used
2016-05-17 22:18:34,036 INFO app.DAGAppMaster (DAGAppMaster.java:run(731)) - Calling stop for all the services
2016-05-17 22:18:34,037 INFO app.DAGAppMaster (DAGAppMaster.java:run(742)) - Exiting DAGAppMaster..GoodBye!
2016-05-17 22:18:34,049 INFO dflt.DefaultSorter (DefaultSorter.java:setEquator(328)) - (EQUATOR) 0 kvi 26214396(104857584)
2016-05-17 22:18:34,049 INFO dflt.DefaultSorter (DefaultSorter.java:<init>(150)) - tez.runtime.io.sort.mb: 100
2016-05-17 22:18:34,049 INFO dflt.DefaultSorter (DefaultSorter.java:<init>(151)) - soft limit at 83886080
2016-05-17 22:18:34,050 INFO dflt.DefaultSorter (DefaultSorter.java:<init>(152)) - bufstart = 0; bufvoid = 104857600
2016-05-17 22:18:34,050 INFO dflt.DefaultSorter (DefaultSorter.java:<init>(153)) - kvstart = 26214396; length = 6553600
2016-05-17 22:18:34,050 INFO element.TezSourceStage (TezSourceStage.java:prepare(63)) - calling MRInput#start() on: Hfs["TextLine[['offset', 'line' | long, String]->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.5.x-tez-0.6.x/build/classes/test/data/lower+upper.txt"]
2016-05-17 22:18:34,050 INFO input.MRInput (MRInput.java:start(450)) - MRInput setup to received events1
2016-05-17 22:18:34,051 INFO tez.FlowProcessor (FlowProcessor.java:waitForInputsReady(202)) - flow node id: AC44F8ABA3AC42CBB216039CD59B227D, all 1 inputs ready in: 00:00:00.000
2016-05-17 22:18:34,053 INFO task.TezTaskRunner (TezTaskRunner.java:run(177)) - Closing task, taskAttemptId=attempt_1463523513603_0001_1_00_000000_0
2016-05-17 22:18:34,053 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: AC44F8ABA3AC42CBB216039CD59B227D, mem on close (mb), free: 463, total: 584, max: 672
2016-05-17 22:18:34,053 INFO dflt.DefaultSorter (DefaultSorter.java:flush(589)) - Starting flush of map output
2016-05-17 22:18:34,053 INFO dflt.DefaultSorter (DefaultSorter.java:flush(607)) - Sorting & Spilling map output
2016-05-17 22:18:34,054 INFO dflt.DefaultSorter (DefaultSorter.java:flush(608)) - bufstart = 0; bufend = 105; bufvoid = 104857600
2016-05-17 22:18:34,054 INFO dflt.DefaultSorter (DefaultSorter.java:flush(610)) - kvstart = 26214396(104857584); kvend = 26214380(104857520); length = 17/6553600
2016-05-17 22:18:34,057 INFO dflt.DefaultSorter (DefaultSorter.java:spill(829)) - Finished spill 0
2016-05-17 22:18:34 Completed running task attempt: attempt_1463523513603_0001_1_00_000000_0
2016-05-17 22:18:34,061 INFO task.TezTaskRunner (TezTaskRunner.java:run(181)) - Task completed, taskAttemptId=attempt_1463523513603_0001_1_00_000000_0, fatalErrorOccurred=false
2016-05-17 22:18:34,062 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1463523513603_0001_1][Event:VERTEX_DATA_MOVEMENT_EVENTS_GENERATED]: vertexId=vertex_1463523513603_0001_1_00, eventCount=1
2016-05-17 22:18:34,062 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:cleanup(712)) - Final Counters : Counters: 27 [[File System Counters FILE_BYTES_READ=77, FILE_BYTES_WRITTEN=157, FILE_READ_OPS=0, FILE_LARGE_READ_OPS=0, FILE_WRITE_OPS=0][org.apache.tez.common.counters.TaskCounter SPILLED_RECORDS=5, GC_TIME_MILLIS=183, CPU_MILLISECONDS=0, PHYSICAL_MEMORY_BYTES=0, VIRTUAL_MEMORY_BYTES=0, COMMITTED_HEAP_BYTES=612368384, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=5, OUTPUT_BYTES=105, OUTPUT_BYTES_WITH_OVERHEAD=121, OUTPUT_BYTES_PHYSICAL=125, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=0, ADDITIONAL_SPILL_COUNT=0][cascading.flow.SliceCounters Process_Begin_Time=1463523514051, Process_Duration=2, Process_End_Time=1463523514053, Read_Duration=0, Tuples_Read=5, Tuples_Written=5, Write_Duration=1][cascading.flow.StepCounters Tuples_Read=5]]
2016-05-17 22:18:34,062 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(164)) - Redirecting log file based on addend:
2016-05-17 22:18:34,062 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(179)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-17 22:18:34,062 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event TASK_ATTEMPT_FINISHED to history file
2016-05-17 22:18:34,062 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1463523513603_0001_1][Event:TASK_ATTEMPT_FINISHED]: vertexName=AC44F8ABA3AC42CBB216039CD59B227D, taskAttemptId=attempt_1463523513603_0001_1_00_000000_0, startTime=1463523513816, finishTime=1463523514062, timeTaken=246, status=SUCCEEDED, errorEnum=, diagnostics=, counters=Counters: 28, org.apache.tez.common.counters.DAGCounter, RACK_LOCAL_TASKS=1, File System Counters, FILE_BYTES_READ=77, FILE_BYTES_WRITTEN=157, FILE_READ_OPS=0, FILE_LARGE_READ_OPS=0, FILE_WRITE_OPS=0, org.apache.tez.common.counters.TaskCounter, SPILLED_RECORDS=5, GC_TIME_MILLIS=183, CPU_MILLISECONDS=0, PHYSICAL_MEMORY_BYTES=0, VIRTUAL_MEMORY_BYTES=0, COMMITTED_HEAP_BYTES=612368384, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=5, OUTPUT_BYTES=105, OUTPUT_BYTES_WITH_OVERHEAD=121, OUTPUT_BYTES_PHYSICAL=125, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=0, ADDITIONAL_SPILL_COUNT=0, cascading.flow.SliceCounters, Process_Begin_Time=1463523514051, Process_Duration=2, Process_End_Time=1463523514053, Read_Duration=0, Tuples_Read=5, Tuples_Written=5, Write_Duration=1, cascading.flow.StepCounters, Tuples_Read=5
2016-05-17 22:18:34,063 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:handle(692)) - attempt_1463523513603_0001_1_00_000000_0 TaskAttempt Transitioned from RUNNING to SUCCEEDED due to event TA_DONE
2016-05-17 22:18:34,063 INFO task.ContainerReporter (ContainerReporter.java:call(55)) - Attempting to fetch new task for container container_1463523513603_0001_00_000001
2016-05-17 22:18:34,063 INFO impl.TaskImpl (TaskImpl.java:transition(1052)) - Task succeeded with attempt attempt_1463523513603_0001_1_00_000000_0
2016-05-17 22:18:34,063 INFO rm.TaskSchedulerEventHandler (TaskSchedulerEventHandler.java:handleEvent(149)) - Processing the event EventType: S_TA_ENDED
2016-05-17 22:18:34,065 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1463523513603_0001_1][Event:TASK_FINISHED]: vertexName=AC44F8ABA3AC42CBB216039CD59B227D, taskId=task_1463523513603_0001_1_00_000000, startTime=1463523513816, finishTime=1463523514064, timeTaken=248, status=SUCCEEDED, successfulAttemptID=attempt_1463523513603_0001_1_00_000000_0, diagnostics=, counters=Counters: 28, org.apache.tez.common.counters.DAGCounter, RACK_LOCAL_TASKS=1, File System Counters, FILE_BYTES_READ=77, FILE_BYTES_WRITTEN=157, FILE_READ_OPS=0, FILE_LARGE_READ_OPS=0, FILE_WRITE_OPS=0, org.apache.tez.common.counters.TaskCounter, SPILLED_RECORDS=5, GC_TIME_MILLIS=183, CPU_MILLISECONDS=0, PHYSICAL_MEMORY_BYTES=0, VIRTUAL_MEMORY_BYTES=0, COMMITTED_HEAP_BYTES=612368384, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=5, OUTPUT_BYTES=105, OUTPUT_BYTES_WITH_OVERHEAD=121, OUTPUT_BYTES_PHYSICAL=125, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=0, ADDITIONAL_SPILL_COUNT=0, cascading.flow.SliceCounters, Process_Begin_Time=1463523514051, Process_Duration=2, Process_End_Time=1463523514053, Read_Duration=0, Tuples_Read=5, Tuples_Written=5, Write_Duration=1, cascading.flow.StepCounters, Tuples_Read=5
2016-05-17 22:18:34,065 INFO impl.TaskImpl (TaskImpl.java:handle(862)) - task_1463523513603_0001_1_00_000000 Task Transitioned from RUNNING to SUCCEEDED due to event T_ATTEMPT_SUCCEEDED
2016-05-17 22:18:34,066 INFO impl.VertexImpl (VertexImpl.java:transition(3536)) - Num completed Tasks for vertex_1463523513603_0001_1_00 [AC44F8ABA3AC42CBB216039CD59B227D] : 1
2016-05-17 22:18:34,066 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event TASK_FINISHED to history file
2016-05-17 22:18:34,066 INFO impl.VertexImpl (VertexImpl.java:checkVertexForCompletion(1674)) - Vertex succeeded: vertex_1463523513603_0001_1_00 [AC44F8ABA3AC42CBB216039CD59B227D]
2016-05-17 22:18:34,068 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event VERTEX_FINISHED to history file
2016-05-17 22:18:34,068 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1463523513603_0001_1][Event:VERTEX_FINISHED]: vertexName=AC44F8ABA3AC42CBB216039CD59B227D, vertexId=vertex_1463523513603_0001_1_00, initRequestedTime=1463523513778, initedTime=1463523513794, startRequestedTime=1463523513799, startedTime=1463523513799, finishTime=1463523514067, timeTaken=268, status=SUCCEEDED, diagnostics=, counters=Counters: 28, org.apache.tez.common.counters.DAGCounter, RACK_LOCAL_TASKS=1, File System Counters, FILE_BYTES_READ=77, FILE_BYTES_WRITTEN=157, FILE_READ_OPS=0, FILE_LARGE_READ_OPS=0, FILE_WRITE_OPS=0, org.apache.tez.common.counters.TaskCounter, SPILLED_RECORDS=5, GC_TIME_MILLIS=183, CPU_MILLISECONDS=0, PHYSICAL_MEMORY_BYTES=0, VIRTUAL_MEMORY_BYTES=0, COMMITTED_HEAP_BYTES=612368384, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=5, OUTPUT_BYTES=105, OUTPUT_BYTES_WITH_OVERHEAD=121, OUTPUT_BYTES_PHYSICAL=125, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=0, ADDITIONAL_SPILL_COUNT=0, cascading.flow.SliceCounters, Process_Begin_Time=1463523514051, Process_Duration=2, Process_End_Time=1463523514053, Read_Duration=0, Tuples_Read=5, Tuples_Written=5, Write_Duration=1, cascading.flow.StepCounters, Tuples_Read=5, vertexStats=firstTaskStartTime=1463523513816, firstTasksToStart=[ task_1463523513603_0001_1_00_000000 ], lastTaskFinishTime=1463523514062, lastTasksToFinish=[ task_1463523513603_0001_1_00_000000 ], minTaskDuration=246, maxTaskDuration=246, avgTaskDuration=246.0, numSuccessfulTasks=1, shortestDurationTasks=[ task_1463523513603_0001_1_00_000000 ], longestDurationTasks=[ task_1463523513603_0001_1_00_000000 ], vertexTaskStats={numFailedTasks=0, numSucceededTasks=1, numKilledTaskAttempts=0, numKilledTasks=0, numFailedTaskAttempts=0, numCompletedTasks=1}
2016-05-17 22:18:34,069 INFO impl.VertexImpl (VertexImpl.java:handle(1564)) - vertex_1463523513603_0001_1_00 [AC44F8ABA3AC42CBB216039CD59B227D] transitioned from RUNNING to SUCCEEDED due to event V_TASK_COMPLETED
2016-05-17 22:18:34,069 INFO container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1463523513603_0001_00_000001 transitioned from RUNNING to IDLE via event C_TA_SUCCEEDED
2016-05-17 22:18:34,069 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1463523513603_0001_1][Event:CONTAINER_STOPPED]: containerId=container_1463523513603_0001_00_000001, stoppedTime=1463523514069, exitStatus=0
2016-05-17 22:18:34,069 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event CONTAINER_STOPPED to history file
2016-05-17 22:18:34,069 INFO container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1463523513603_0001_00_000001 transitioned from IDLE to STOP_REQUESTED via event C_STOP_REQUEST
2016-05-17 22:18:34,069 INFO impl.VertexImpl (VertexImpl.java:transition(3477)) - Source task attempt completed for vertex: vertex_1463523513603_0001_1_01 [A95EBDDFBC0344098A63024BC7984A3D] attempt: attempt_1463523513603_0001_1_00_000000_0 with state: SUCCEEDED vertexState: RUNNING
2016-05-17 22:18:34,070 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:schedulePendingTasks(608)) - All source tasks assigned. Ramping up 1 remaining tasks for vertex: A95EBDDFBC0344098A63024BC7984A3D
2016-05-17 22:18:34,070 INFO impl.VertexImpl (VertexImpl.java:scheduleTasks(1250)) - Routing pending task events for vertex: vertex_1463523513603_0001_1_01 [A95EBDDFBC0344098A63024BC7984A3D]
2016-05-17 22:18:34,070 INFO impl.DAGImpl (DAGImpl.java:transition(1747)) - Vertex vertex_1463523513603_0001_1_00 [AC44F8ABA3AC42CBB216039CD59B227D] completed., numCompletedVertices=1, numSuccessfulVertices=1, numFailedVertices=0, numKilledVertices=0, numVertices=2
2016-05-17 22:18:34,070 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:stop(244)) - Ignoring stop request for containerId container_1463523513603_0001_00_000001. Relying on regular task shutdown for it to end
2016-05-17 22:18:34,071 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event TASK_STARTED to history file
2016-05-17 22:18:34,071 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1463523513603_0001_1][Event:TASK_STARTED]: vertexName=A95EBDDFBC0344098A63024BC7984A3D, taskId=task_1463523513603_0001_1_01_000000, scheduledTime=1463523514070, launchTime=1463523514070
2016-05-17 22:18:34,071 INFO impl.TaskImpl (TaskImpl.java:handle(862)) - task_1463523513603_0001_1_01_000000 Task Transitioned from NEW to SCHEDULED due to event T_SCHEDULE
2016-05-17 22:18:34,071 INFO container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1463523513603_0001_00_000001 transitioned from STOP_REQUESTED to STOPPING via event C_NM_STOP_SENT
2016-05-17 22:18:34,071 INFO impl.DAGSchedulerNaturalOrderControlled (DAGSchedulerNaturalOrderControlled.java:scheduleTask(111)) - Scheduled vertex: vertex_1463523513603_0001_1_01 [A95EBDDFBC0344098A63024BC7984A3D]
2016-05-17 22:18:34,072 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:transition(1014)) - remoteTaskSpec:DAGName : [86EF4021943745A2BAAB6F46343D991A/E27B02069D014C1E84A753853F2C4529] test/(1/1) ...testSimpleBuffer3/simple3, VertexName: A95EBDDFBC0344098A63024BC7984A3D, VertexParallelism: 1, TaskAttemptID:attempt_1463523513603_0001_1_01_000000_0, processorName=cascading.flow.tez.FlowProcessor, inputSpecListSize=1, outputSpecListSize=1, inputSpecList=[{{ sourceVertexName=AC44F8ABA3AC42CBB216039CD59B227D, physicalEdgeCount=1, inputClassName=org.apache.tez.runtime.library.input.OrderedGroupedKVInput }}, ], outputSpecList=[{{ destinationVertexName=B6674A59136C4FE092506740C4EA5E16, physicalEdgeCount=0, outputClassName=org.apache.tez.mapreduce.output.MROutput }}, ]
2016-05-17 22:18:34,072 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:handle(692)) - attempt_1463523513603_0001_1_01_000000_0 TaskAttempt Transitioned from NEW to START_WAIT due to event TA_SCHEDULE
2016-05-17 22:18:34,072 INFO rm.TaskSchedulerEventHandler (TaskSchedulerEventHandler.java:handleEvent(149)) - Processing the event EventType: S_TA_LAUNCH_REQUEST
2016-05-17 22:18:34,073 INFO container.AMContainerImpl (AMContainerImpl.java:transition(521)) - Sending Launch Request for Container with id: container_1463523513603_0001_00_000002
2016-05-17 22:18:34,073 INFO container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1463523513603_0001_00_000002 transitioned from ALLOCATED to LAUNCHING via event C_LAUNCH_REQUEST
2016-05-17 22:18:34,074 INFO task.TezChild (TezChild.java:newTezChild(432)) - PID, containerIdentifier: null, container_1463523513603_0001_00_000002
2016-05-17 22:18:34,075 WARN impl.MetricsSystemImpl (MetricsSystemImpl.java:init(145)) - DAGAppMaster metrics system already initialized!
2016-05-17 22:18:34,075 INFO app.TaskAttemptListenerImpTezDag (TaskAttemptListenerImpTezDag.java:getTask(213)) - Container with id: container_1463523513603_0001_00_000001 is valid, but no longer registered, and will be killed
2016-05-17 22:18:34,075 INFO task.ContainerReporter (ContainerReporter.java:call(65)) - Got TaskUpdate for containerId= container_1463523513603_0001_00_000001: 11 ms after starting to poll. TaskInfo: shouldDie: true
2016-05-17 22:18:34,075 INFO task.TezChild (TezChild.java:run(207)) - ContainerTask returned shouldDie=true for container container_1463523513603_0001_00_000001, Exiting
2016-05-17 22:18:34,075 INFO task.TezChild (TezChild.java:shutdown(343)) - Shutdown invoked for container container_1463523513603_0001_00_000001
2016-05-17 22:18:34,075 INFO task.TezChild (TezChild.java:shutdown(345)) - Shutting down container container_1463523513603_0001_00_000001
2016-05-17 22:18:34,077 INFO container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1463523513603_0001_00_000002 transitioned from LAUNCHING to IDLE via event C_LAUNCHED
2016-05-17 22:18:34,077 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event CONTAINER_LAUNCHED to history file
2016-05-17 22:18:34,077 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(270)) - ContainerExecutionResult for: container_1463523513603_0001_00_000001 = ContainerExecutionResult{exitStatus=SUCCESS, throwable=null, errorMessage='Asked to die by the AM'}
2016-05-17 22:18:34,077 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1463523513603_0001_1][Event:CONTAINER_LAUNCHED]: containerId=container_1463523513603_0001_00_000002, launchTime=1463523514077
2016-05-17 22:18:34,078 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(274)) - Container: container_1463523513603_0001_00_000001 completed successfully
2016-05-17 22:18:34,078 INFO container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1463523513603_0001_00_000001 transitioned from STOPPING to COMPLETED via event C_COMPLETED
2016-05-17 22:18:34,079 INFO task.ContainerReporter (ContainerReporter.java:call(55)) - Attempting to fetch new task for container container_1463523513603_0001_00_000002
2016-05-17 22:18:34,079 INFO container.AMContainerImpl (AMContainerImpl.java:transition(827)) - Assigned taskAttempt + [attempt_1463523513603_0001_1_01_000000_0] to container: [container_1463523513603_0001_00_000002]
2016-05-17 22:18:34,079 INFO container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1463523513603_0001_00_000002 transitioned from IDLE to RUNNING via event C_PULL_TA
2016-05-17 22:18:34,080 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:transition(1146)) - TaskAttempt: [attempt_1463523513603_0001_1_01_000000_0] started. Is using containerId: [container_1463523513603_0001_00_000002] on NM: [127.0.0.1:0]
2016-05-17 22:18:34,080 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1463523513603_0001_1][Event:TASK_ATTEMPT_STARTED]: vertexName=A95EBDDFBC0344098A63024BC7984A3D, taskAttemptId=attempt_1463523513603_0001_1_01_000000_0, startTime=1463523514080, containerId=container_1463523513603_0001_00_000002, nodeId=127.0.0.1:0, inProgressLogs=127.0.0.1:0/node/containerlogs/container_1463523513603_0001_00_000002/teamcity, completedLogs=
2016-05-17 22:18:34,080 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event TASK_ATTEMPT_STARTED to history file
2016-05-17 22:18:34,080 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:handle(692)) - attempt_1463523513603_0001_1_01_000000_0 TaskAttempt Transitioned from START_WAIT to RUNNING due to event TA_STARTED_REMOTELY
2016-05-17 22:18:34,080 INFO impl.TaskImpl (TaskImpl.java:handle(862)) - task_1463523513603_0001_1_01_000000 Task Transitioned from SCHEDULED to RUNNING due to event T_ATTEMPT_LAUNCHED
2016-05-17 22:18:34,081 INFO app.TaskAttemptListenerImpTezDag (TaskAttemptListenerImpTezDag.java:getTask(240)) - Container with id: container_1463523513603_0001_00_000002 given task: attempt_1463523513603_0001_1_01_000000_0
2016-05-17 22:18:34,081 INFO task.ContainerReporter (ContainerReporter.java:call(65)) - Got TaskUpdate for containerId= container_1463523513603_0001_00_000002: 0 ms after starting to poll. TaskInfo: shouldDie: false, currentTaskAttemptId: attempt_1463523513603_0001_1_01_000000_0
2016-05-17 22:18:34 Starting to run new task attempt: attempt_1463523513603_0001_1_01_000000_0
2016-05-17 22:18:34,082 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(164)) - Redirecting log file based on addend: attempt_1463523513603_0001_1_01_000000_0
2016-05-17 22:18:34,082 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(179)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-17 22:18:34,082 INFO task.TezChild (TezChild.java:handleNewTaskCredentials(273)) - Refreshing UGI since Credentials have changed
2016-05-17 22:18:34,082 INFO task.TezChild (TezChild.java:handleNewTaskCredentials(276)) - Credentials : #Tokens=1, #SecretKeys=0
2016-05-17 22:18:34,083 INFO task.TezChild (TezChild.java:handleNewTaskLocalResources(302)) - Localizing additional local resources for Task : {}
2016-05-17 22:18:34,083 INFO task.TezChild (TezChild.java:handleNewTaskLocalResources(312)) - Done localizing additional resources
2016-05-17 22:18:34,083 INFO metrics.TaskCounterUpdater (TaskCounterUpdater.java:initResourceCalculatorPlugin(148)) - Using ResourceCalculatorProcessTree : [ ]
2016-05-17 22:18:34,083 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:<init>(149)) - Initializing LogicalIOProcessorRuntimeTask with TaskSpec: DAGName : [86EF4021943745A2BAAB6F46343D991A/E27B02069D014C1E84A753853F2C4529] test/(1/1) ...testSimpleBuffer3/simple3, VertexName: A95EBDDFBC0344098A63024BC7984A3D, VertexParallelism: 1, TaskAttemptID:attempt_1463523513603_0001_1_01_000000_0, processorName=cascading.flow.tez.FlowProcessor, inputSpecListSize=1, outputSpecListSize=1, inputSpecList=[{{ sourceVertexName=AC44F8ABA3AC42CBB216039CD59B227D, physicalEdgeCount=1, inputClassName=org.apache.tez.runtime.library.input.OrderedGroupedKVInput }}, ], outputSpecList=[{{ destinationVertexName=B6674A59136C4FE092506740C4EA5E16, physicalEdgeCount=0, outputClassName=org.apache.tez.mapreduce.output.MROutput }}, ]
2016-05-17 22:18:34,084 INFO resources.MemoryDistributor (MemoryDistributor.java:<init>(85)) - InitialMemoryDistributor (isEnabled=true) invoked with: numInputs=1, numOutputs=1, JVM.maxFree=704643072
2016-05-17 22:18:34,088 INFO task.TezTaskRunner (TezTaskRunner.java:run(172)) - Initializing task, taskAttemptId=attempt_1463523513603_0001_1_01_000000_0
2016-05-17 22:18:34,089 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(191)) - Initializing LogicalProcessorIORuntimeTask
2016-05-17 22:18:34,088 INFO impl.TaskImpl (TaskImpl.java:getTaskAttemptTezEvents(481)) - TaskAttempt:attempt_1463523513603_0001_1_01_000000_0 sent events: (0-1)
2016-05-17 22:18:34,089 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(195)) - Creating processor, processorClassName=cascading.flow.tez.FlowProcessor
2016-05-17 22:18:34,090 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(389)) - Initializing Input using InputSpec: { sourceVertexName=AC44F8ABA3AC42CBB216039CD59B227D, physicalEdgeCount=1, inputClassName=org.apache.tez.runtime.library.input.OrderedGroupedKVInput }
2016-05-17 22:18:34,090 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:createInput(537)) - Creating Input
2016-05-17 22:18:34,091 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(397)) - Initializing Input with src edge: AC44F8ABA3AC42CBB216039CD59B227D
2016-05-17 22:18:34,091 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initializeLogicalIOProcessor(488)) - Initializing processor, processorClassName=cascading.flow.tez.FlowProcessor
2016-05-17 22:18:34,092 INFO orderedgrouped.MergeManager (MergeManager.java:getInitialMemoryRequirement(310)) - Initial Shuffle Memory Required: 634178752, based on INPUT_BUFFER_factor: 0.9
2016-05-17 22:18:34,092 INFO orderedgrouped.MergeManager (MergeManager.java:getInitialMemoryRequirement(320)) - Initial Memory required for final merged output: 0, using factor: 0.0
2016-05-17 22:18:34,091 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(438)) - Initializing Output using OutputSpec: { destinationVertexName=B6674A59136C4FE092506740C4EA5E16, physicalEdgeCount=0, outputClassName=org.apache.tez.mapreduce.output.MROutput }
2016-05-17 22:18:34,093 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:createOutput(565)) - Creating Output
2016-05-17 22:18:34,094 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(446)) - Initializing Output with dest edge: B6674A59136C4FE092506740C4EA5E16
2016-05-17 22:18:34,094 INFO output.MROutput (MROutput.java:initialize(347)) - Initializing Simple Output
2016-05-17 22:18:34,094 INFO resources.MemoryDistributor$RequestorInfo (MemoryDistributor.java:<init>(233)) - Received request: 634178752, type: INPUT, componentVertexName: AC44F8ABA3AC42CBB216039CD59B227D
2016-05-17 22:18:34,095 INFO tez.FlowProcessor (FlowProcessor.java:initialize(88)) - cascading version: 3.1.0
2016-05-17 22:18:34,094 INFO resources.MemoryDistributor$RequestorInfo (MemoryDistributor.java:<init>(233)) - Received request: 0, type: OUTPUT, componentVertexName: B6674A59136C4FE092506740C4EA5E16
2016-05-17 22:18:34,095 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(403)) - Initialized Input with src edge: AC44F8ABA3AC42CBB216039CD59B227D
2016-05-17 22:18:34,103 INFO tez.FlowProcessor (FlowProcessor.java:initialize(94)) - flow node id: A95EBDDFBC0344098A63024BC7984A3D, ordinal: 1
2016-05-17 22:18:34,103 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: A95EBDDFBC0344098A63024BC7984A3D, mem on start (mb), free: 457, total: 584, max: 672
2016-05-17 22:18:34,103 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initializeLogicalIOProcessor(491)) - Initialized processor, processorClassName=cascading.flow.tez.FlowProcessor
2016-05-17 22:18:34,103 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(220)) - Waiting for 2 initializers to finish
2016-05-17 22:18:34,103 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(220)) - Waiting for 1 initializers to finish
2016-05-17 22:18:34,105 INFO output.MROutput (MROutput.java:initialize(418)) - Initialized Simple Output, using_new_api: false
2016-05-17 22:18:34,106 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(452)) - Initialized Output with dest edge: B6674A59136C4FE092506740C4EA5E16
2016-05-17 22:18:34,106 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(233)) - All initializers finished
2016-05-17 22:18:34,106 INFO resources.MemoryDistributor (MemoryDistributor.java:makeInitialAllocations(124)) - Using Allocator class: org.apache.tez.runtime.library.resources.WeightedScalingMemoryDistributor
2016-05-17 22:18:34,106 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:initialProcessMemoryRequestContext(171)) - ScaleFactor: 12, for type: SORTED_MERGED_INPUT
2016-05-17 22:18:34,106 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:getRequestTypeForClass(197)) - Falling back to RequestType.OTHER for class: org.apache.tez.mapreduce.output.MROutput
2016-05-17 22:18:34,106 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:initialProcessMemoryRequestContext(171)) - ScaleFactor: 1, for type: OTHER
2016-05-17 22:18:34,107 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:computeReservedFraction(262)) - InitialReservationFraction=0.3, AdditionalReservationFractionForIOs=0.03, finalReserveFractionUsed=0.32999999999999996
2016-05-17 22:18:34,107 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:assignMemory(126)) - Scaling Requests. NumRequests: 2, numScaledRequests: 13, TotalRequested: 634178752, TotalRequestedScaled: 5.853957710769231E8, TotalJVMHeap: 704643072, TotalAvailable: 472110858, TotalRequested/TotalJVMHeap:0.90
2016-05-17 22:18:34,107 INFO resources.MemoryDistributor (MemoryDistributor.java:makeInitialAllocations(138)) - Informing: INPUT, AC44F8ABA3AC42CBB216039CD59B227D, org.apache.tez.runtime.library.input.OrderedGroupedKVInput: requested=634178752, allocated=472110858
2016-05-17 22:18:34,107 INFO resources.MemoryDistributor (MemoryDistributor.java:makeInitialAllocations(138)) - Informing: OUTPUT, B6674A59136C4FE092506740C4EA5E16, org.apache.tez.mapreduce.output.MROutput: requested=0, allocated=0
2016-05-17 22:18:34,107 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(255)) - Starting Inputs/Outputs
2016-05-17 22:18:34,107 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(268)) - Input: AC44F8ABA3AC42CBB216039CD59B227D being auto started by the framework. Subsequent instances will not be auto-started
2016-05-17 22:18:34,107 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(289)) - Num IOs determined for AutoStart: 1
2016-05-17 22:18:34,107 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(291)) - Waiting for 1 IOs to start
2016-05-17 22:18:34,108 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(419)) - Starting Input with src edge: AC44F8ABA3AC42CBB216039CD59B227D
2016-05-17 22:18:34,109 INFO common.TezRuntimeUtils (TezRuntimeUtils.java:instantiateCombiner(65)) - No combiner specified via tez.runtime.combiner.class. Combiner will not be used
2016-05-17 22:18:34,109 INFO orderedgrouped.Shuffle (Shuffle.java:<init>(182)) - Shuffle assigned with 1 inputs, codec: NoneifileReadAhead: true
2016-05-17 22:18:34,109 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:<init>(158)) - ShuffleScheduler running for sourceVertex: AC44F8ABA3AC42CBB216039CD59B227D with configuration: maxFetchFailuresBeforeReporting=5, reportReadErrorImmediately=true, maxFailedUniqueFetches=1, abortFailureLimit=30, maxMapRuntime=0, maxTaskOutputAtOnce=20
2016-05-17 22:18:34,110 INFO orderedgrouped.MergeManager (MergeManager.java:<init>(221)) - InitialRequest: ShuffleMem=634178752, postMergeMem=0, RuntimeTotalAvailable=472110858. Updated to: ShuffleMem=472110858, postMergeMem: 0
2016-05-17 22:18:34,110 INFO orderedgrouped.MergeManager (MergeManager.java:<init>(252)) - MergerManager: memoryLimit=472110858, maxSingleShuffleLimit=118027712, mergeThreshold=424899744, ioSortFactor=100, memToMemMergeOutputsThreshold=100
2016-05-17 22:18:34,110 INFO orderedgrouped.Shuffle (Shuffle.java:<init>(233)) - Num fetchers being started: 1
2016-05-17 22:18:34,111 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(421)) - Started Input with src edge: AC44F8ABA3AC42CBB216039CD59B227D
2016-05-17 22:18:34,113 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(304)) - AutoStartComplete
2016-05-17 22:18:34,114 INFO task.TezTaskRunner (TezTaskRunner.java:run(175)) - Running task, taskAttemptId=attempt_1463523513603_0001_1_01_000000_0
2016-05-17 22:18:34,115 INFO orderedgrouped.ShuffleInputEventHandlerOrderedGrouped (ShuffleInputEventHandlerOrderedGrouped.java:processDataMovementEvent(84)) - DME srcIdx: 0, targetIdx: 0, attemptNum: 0, payload: [host: ip-10-123-148-208, port: 0, pathComponent: attempt_1463523513603_0001_1_00_000000_0_10036, runDuration: 228653, ]
2016-05-17 22:18:34,117 INFO graph.Hadoop2TezStreamGraph (Hadoop2TezStreamGraph.java:buildGraph(151)) - using streamed source: GroupBy(test)[by:[{1}:'num']]
2016-05-17 22:18:34,118 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:logIndividualFetchComplete(250)) - Completed fetch for attempt: InputAttemptIdentifier [inputIdentifier=InputIdentifier [inputIndex=0], attemptNumber=0, pathComponent=attempt_1463523513603_0001_1_00_000000_0_10036] to DISK_DIRECT, CompressedSize=125, DecompressedSize=121,EndTime=1463523514117, TimeTaken=1, Rate=0.12 MB/s
2016-05-17 22:18:34,118 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:copySucceeded(213)) - All inputs fetched for input vertex : AC44F8ABA3AC42CBB216039CD59B227D
2016-05-17 22:18:34,118 INFO tez.FlowProcessor (FlowProcessor.java:run(121)) - sourcing from: GroupBy(test)[by:[{1}:'num']] streamed: true, id: 5DBC737D09BF4B5EB74DE9B1EF8311A7
2016-05-17 22:18:34,118 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:logProgress(263)) - copy(1 of 1. Transfer rate (CumulativeDataFetched/TimeSinceInputStarted)) 0.00 MB/s)
2016-05-17 22:18:34,118 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:freeHost(529)) - ip-10-123-148-208:0 freed by fetcher [AC44F8ABA3AC42CBB216039CD59B227D] #7 in 2ms
2016-05-17 22:18:34,118 INFO tez.FlowProcessor (FlowProcessor.java:run(124)) - sinking to: Hfs["TextLine[['line']->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.5.x-tez-0.6.x/build/test/output/hadoop2-tez/bufferpipesplatform/testSimpleBuffer3/simple3"], id: B6674A59136C4FE092506740C4EA5E16
2016-05-17 22:18:34,119 INFO orderedgrouped.Shuffle (Shuffle.java:cleanupFetchers(394)) - Shutdown..fetcher [AC44F8ABA3AC42CBB216039CD59B227D] #7, status:false, isInterrupted:false
2016-05-17 22:18:34,120 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(854)) - finalMerge called with 0 in-memory map-outputs and 1 on-disk map-outputs
2016-05-17 22:18:34,120 INFO element.TezSinkStage (TezSinkStage.java:prepare(57)) - calling MROutput#start() on: Hfs["TextLine[['line']->[ALL]]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.5.x-tez-0.6.x/build/test/output/hadoop2-tez/bufferpipesplatform/testSimpleBuffer3/simple3"]
2016-05-17 22:18:34,120 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(951)) - Merging 1 files, 125 bytes from disk
2016-05-17 22:18:34,120 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(966)) - Merging 0 segments, 0 bytes from memory into reduce
2016-05-17 22:18:34,121 INFO impl.TezMerger (TezMerger.java:merge(560)) - Merging 1 sorted segments
2016-05-17 22:18:34,120 INFO element.TezGroupGate (TezGroupGate.java:prepare(90)) - calling OrderedGroupedKVInput#start() on: GroupBy(test)[by:[{1}:'num']] 5DBC737D09BF4B5EB74DE9B1EF8311A7, for 1 inputs
2016-05-17 22:18:34,121 INFO impl.TezMerger (TezMerger.java:merge(665)) - Down to the last merge-pass, with 1 segments left of total size: 108 bytes
2016-05-17 22:18:34,121 INFO orderedgrouped.Shuffle (Shuffle.java:call(381)) - merge complete for input vertex : AC44F8ABA3AC42CBB216039CD59B227D
2016-05-17 22:18:34,122 INFO tez.FlowProcessor (FlowProcessor.java:waitForInputsReady(202)) - flow node id: A95EBDDFBC0344098A63024BC7984A3D, all 1 inputs ready in: 00:00:00.000
2016-05-17 22:18:34,122 INFO orderedgrouped.Shuffle (Shuffle.java:onSuccess(488)) - Shuffle Runner thread complete
2016-05-17 22:18:34,123 INFO common.ValuesIterator (ValuesIterator.java:<init>(83)) - keyDeserializer=cascading.tuple.hadoop.io.KeyTupleDeserializer@9030ca2; valueDeserializer=cascading.tuple.hadoop.io.ValueTupleDeserializer@2732644d
2016-05-17 22:18:34,125 INFO output.MROutput (MROutput.java:flush(534)) - Flushing Simple Output
2016-05-17 22:18:34,126 INFO output.MROutput (MROutput.java:flush(544)) - Flushed Simple Output
2016-05-17 22:18:34,126 INFO app.TaskAttemptListenerImpTezDag (TaskAttemptListenerImpTezDag.java:canCommit(353)) - Commit go/no-go request from attempt_1463523513603_0001_1_01_000000_0
2016-05-17 22:18:34,126 INFO impl.TaskImpl (TaskImpl.java:canCommit(750)) - attempt_1463523513603_0001_1_01_000000_0 given a go for committing the task output.
2016-05-17 22:18:34,127 INFO task.TezTaskRunner (TezTaskRunner.java:run(177)) - Closing task, taskAttemptId=attempt_1463523513603_0001_1_01_000000_0
2016-05-17 22:18:34,127 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: A95EBDDFBC0344098A63024BC7984A3D, mem on close (mb), free: 454, total: 584, max: 672
2016-05-17 22:18:34,127 INFO orderedgrouped.Shuffle (Shuffle.java:shutdown(320)) - Shutting down Shuffle for source: AC44F8ABA3AC42CBB216039CD59B227D
2016-05-17 22:18:34,129 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(854)) - finalMerge called with 0 in-memory map-outputs and 0 on-disk map-outputs
2016-05-17 22:18:34,129 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(951)) - Merging 0 files, 0 bytes from disk
2016-05-17 22:18:34,129 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(966)) - Merging 0 segments, 0 bytes from memory into reduce
2016-05-17 22:18:34,129 INFO impl.TezMerger (TezMerger.java:merge(560)) - Merging 0 sorted segments
2016-05-17 22:18:34,130 INFO impl.TezMerger (TezMerger.java:merge(562)) - Nothing to merge. Returning an empty iterator
2016-05-17 22:18:34 Completed running task attempt: attempt_1463523513603_0001_1_01_000000_0
2016-05-17 22:18:34,131 INFO task.TezTaskRunner (TezTaskRunner.java:run(181)) - Task completed, taskAttemptId=attempt_1463523513603_0001_1_01_000000_0, fatalErrorOccurred=false
2016-05-17 22:18:34,132 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event TASK_ATTEMPT_FINISHED to history file
2016-05-17 22:18:34,132 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1463523513603_0001_1][Event:TASK_ATTEMPT_FINISHED]: vertexName=A95EBDDFBC0344098A63024BC7984A3D, taskAttemptId=attempt_1463523513603_0001_1_01_000000_0, startTime=1463523514080, finishTime=1463523514131, timeTaken=51, status=SUCCEEDED, errorEnum=, diagnostics=, counters=Counters: 46, File System Counters, FILE_BYTES_READ=157, FILE_BYTES_WRITTEN=52, FILE_READ_OPS=0, FILE_LARGE_READ_OPS=0, FILE_WRITE_OPS=0, org.apache.tez.common.counters.TaskCounter, REDUCE_INPUT_GROUPS=4, REDUCE_INPUT_RECORDS=5, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=5, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, GC_TIME_MILLIS=0, CPU_MILLISECONDS=0, PHYSICAL_MEMORY_BYTES=0, VIRTUAL_MEMORY_BYTES=0, COMMITTED_HEAP_BYTES=612368384, OUTPUT_RECORDS=5, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=125, SHUFFLE_BYTES=125, SHUFFLE_BYTES_DECOMPRESSED=121, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=125, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=9, MERGE_PHASE_TIME=12, FIRST_EVENT_RECEIVED=7, LAST_EVENT_RECEIVED=7, Shuffle Errors, BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0, cascading.flow.SliceCounters, Process_Begin_Time=1463523514122, Process_Duration=5, Process_End_Time=1463523514127, Read_Duration=0, Tuples_Read=5, Tuples_Written=5, Write_Duration=0, cascading.flow.StepCounters, Tuples_Written=5
2016-05-17 22:18:34,132 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:handle(692)) - attempt_1463523513603_0001_1_01_000000_0 TaskAttempt Transitioned from RUNNING to SUCCEEDED due to event TA_DONE
2016-05-17 22:18:34,132 INFO rm.TaskSchedulerEventHandler (TaskSchedulerEventHandler.java:handleEvent(149)) - Processing the event EventType: S_TA_ENDED
2016-05-17 22:18:34,132 INFO impl.TaskImpl (TaskImpl.java:transition(1052)) - Task succeeded with attempt attempt_1463523513603_0001_1_01_000000_0
2016-05-17 22:18:34,133 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event TASK_FINISHED to history file
2016-05-17 22:18:34,133 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1463523513603_0001_1][Event:TASK_FINISHED]: vertexName=A95EBDDFBC0344098A63024BC7984A3D, taskId=task_1463523513603_0001_1_01_000000, startTime=1463523514080, finishTime=1463523514133, timeTaken=53, status=SUCCEEDED, successfulAttemptID=attempt_1463523513603_0001_1_01_000000_0, diagnostics=, counters=Counters: 46, File System Counters, FILE_BYTES_READ=157, FILE_BYTES_WRITTEN=52, FILE_READ_OPS=0, FILE_LARGE_READ_OPS=0, FILE_WRITE_OPS=0, org.apache.tez.common.counters.TaskCounter, REDUCE_INPUT_GROUPS=4, REDUCE_INPUT_RECORDS=5, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=5, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, GC_TIME_MILLIS=0, CPU_MILLISECONDS=0, PHYSICAL_MEMORY_BYTES=0, VIRTUAL_MEMORY_BYTES=0, COMMITTED_HEAP_BYTES=612368384, OUTPUT_RECORDS=5, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=125, SHUFFLE_BYTES=125, SHUFFLE_BYTES_DECOMPRESSED=121, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=125, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=9, MERGE_PHASE_TIME=12, FIRST_EVENT_RECEIVED=7, LAST_EVENT_RECEIVED=7, Shuffle Errors, BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0, cascading.flow.SliceCounters, Process_Begin_Time=1463523514122, Process_Duration=5, Process_End_Time=1463523514127, Read_Duration=0, Tuples_Read=5, Tuples_Written=5, Write_Duration=0, cascading.flow.StepCounters, Tuples_Written=5
2016-05-17 22:18:34,134 INFO impl.TaskImpl (TaskImpl.java:handle(862)) - task_1463523513603_0001_1_01_000000 Task Transitioned from RUNNING to SUCCEEDED due to event T_ATTEMPT_SUCCEEDED
2016-05-17 22:18:34,134 INFO impl.VertexImpl (VertexImpl.java:transition(3536)) - Num completed Tasks for vertex_1463523513603_0001_1_01 [A95EBDDFBC0344098A63024BC7984A3D] : 1
2016-05-17 22:18:34,134 INFO impl.VertexImpl (VertexImpl.java:checkVertexForCompletion(1674)) - Vertex succeeded: vertex_1463523513603_0001_1_01 [A95EBDDFBC0344098A63024BC7984A3D]
2016-05-17 22:18:34,135 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event VERTEX_FINISHED to history file
2016-05-17 22:18:34,135 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1463523513603_0001_1][Event:VERTEX_FINISHED]: vertexName=A95EBDDFBC0344098A63024BC7984A3D, vertexId=vertex_1463523513603_0001_1_01, initRequestedTime=1463523513780, initedTime=1463523513790, startRequestedTime=1463523513803, startedTime=1463523513803, finishTime=1463523514134, timeTaken=331, status=SUCCEEDED, diagnostics=, counters=Counters: 46, File System Counters, FILE_BYTES_READ=157, FILE_BYTES_WRITTEN=52, FILE_READ_OPS=0, FILE_LARGE_READ_OPS=0, FILE_WRITE_OPS=0, org.apache.tez.common.counters.TaskCounter, REDUCE_INPUT_GROUPS=4, REDUCE_INPUT_RECORDS=5, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=5, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, GC_TIME_MILLIS=0, CPU_MILLISECONDS=0, PHYSICAL_MEMORY_BYTES=0, VIRTUAL_MEMORY_BYTES=0, COMMITTED_HEAP_BYTES=612368384, OUTPUT_RECORDS=5, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=125, SHUFFLE_BYTES=125, SHUFFLE_BYTES_DECOMPRESSED=121, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=125, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=9, MERGE_PHASE_TIME=12, FIRST_EVENT_RECEIVED=7, LAST_EVENT_RECEIVED=7, Shuffle Errors, BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0, cascading.flow.SliceCounters, Process_Begin_Time=1463523514122, Process_Duration=5, Process_End_Time=1463523514127, Read_Duration=0, Tuples_Read=5, Tuples_Written=5, Write_Duration=0, cascading.flow.StepCounters, Tuples_Written=5, vertexStats=firstTaskStartTime=1463523514080, firstTasksToStart=[ task_1463523513603_0001_1_01_000000 ], lastTaskFinishTime=1463523514131, lastTasksToFinish=[ task_1463523513603_0001_1_01_000000 ], minTaskDuration=51, maxTaskDuration=51, avgTaskDuration=51.0, numSuccessfulTasks=1, shortestDurationTasks=[ task_1463523513603_0001_1_01_000000 ], longestDurationTasks=[ task_1463523513603_0001_1_01_000000 ], vertexTaskStats={numFailedTasks=0, numSucceededTasks=1, numKilledTaskAttempts=0, numKilledTasks=0, numFailedTaskAttempts=0, numCompletedTasks=1}
2016-05-17 22:18:34,136 INFO impl.VertexImpl (VertexImpl.java:handle(1564)) - vertex_1463523513603_0001_1_01 [A95EBDDFBC0344098A63024BC7984A3D] transitioned from RUNNING to SUCCEEDED due to event V_TASK_COMPLETED
2016-05-17 22:18:34,136 INFO container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1463523513603_0001_00_000002 transitioned from RUNNING to IDLE via event C_TA_SUCCEEDED
2016-05-17 22:18:34,136 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:cleanup(712)) - Final Counters : Counters: 46 [[File System Counters FILE_BYTES_READ=157, FILE_BYTES_WRITTEN=52, FILE_READ_OPS=0, FILE_LARGE_READ_OPS=0, FILE_WRITE_OPS=0][org.apache.tez.common.counters.TaskCounter REDUCE_INPUT_GROUPS=4, REDUCE_INPUT_RECORDS=5, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=5, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, GC_TIME_MILLIS=0, CPU_MILLISECONDS=0, PHYSICAL_MEMORY_BYTES=0, VIRTUAL_MEMORY_BYTES=0, COMMITTED_HEAP_BYTES=612368384, OUTPUT_RECORDS=5, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=125, SHUFFLE_BYTES=125, SHUFFLE_BYTES_DECOMPRESSED=121, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=125, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=9, MERGE_PHASE_TIME=12, FIRST_EVENT_RECEIVED=7, LAST_EVENT_RECEIVED=7][Shuffle Errors BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0][cascading.flow.SliceCounters Process_Begin_Time=1463523514122, Process_Duration=5, Process_End_Time=1463523514127, Read_Duration=0, Tuples_Read=5, Tuples_Written=5, Write_Duration=0][cascading.flow.StepCounters Tuples_Written=5]]
2016-05-17 22:18:34,137 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event CONTAINER_STOPPED to history file
2016-05-17 22:18:34,137 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(164)) - Redirecting log file based on addend:
2016-05-17 22:18:34,137 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(179)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-17 22:18:34,136 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1463523513603_0001_1][Event:CONTAINER_STOPPED]: containerId=container_1463523513603_0001_00_000002, stoppedTime=1463523514136, exitStatus=0
2016-05-17 22:18:34,137 INFO task.ContainerReporter (ContainerReporter.java:call(55)) - Attempting to fetch new task for container container_1463523513603_0001_00_000002
2016-05-17 22:18:34,137 INFO container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1463523513603_0001_00_000002 transitioned from IDLE to STOP_REQUESTED via event C_STOP_REQUEST
2016-05-17 22:18:34,138 INFO app.TaskAttemptListenerImpTezDag (TaskAttemptListenerImpTezDag.java:getTask(213)) - Container with id: container_1463523513603_0001_00_000002 is valid, but no longer registered, and will be killed
2016-05-17 22:18:34,138 INFO impl.DAGImpl (DAGImpl.java:transition(1747)) - Vertex vertex_1463523513603_0001_1_01 [A95EBDDFBC0344098A63024BC7984A3D] completed., numCompletedVertices=2, numSuccessfulVertices=2, numFailedVertices=0, numKilledVertices=0, numVertices=2
2016-05-17 22:18:34,138 INFO task.ContainerReporter (ContainerReporter.java:call(65)) - Got TaskUpdate for containerId= container_1463523513603_0001_00_000002: 0 ms after starting to poll. TaskInfo: shouldDie: true
2016-05-17 22:18:34,138 INFO impl.DAGImpl (DAGImpl.java:commitOrAbortOutputs(831)) - Calling DAG commit/abort for dag: dag_1463523513603_0001_1
2016-05-17 22:18:34,138 INFO task.TezChild (TezChild.java:run(207)) - ContainerTask returned shouldDie=true for container container_1463523513603_0001_00_000002, Exiting
2016-05-17 22:18:34,139 INFO task.TezChild (TezChild.java:shutdown(343)) - Shutdown invoked for container container_1463523513603_0001_00_000002
2016-05-17 22:18:34,139 INFO task.TezChild (TezChild.java:shutdown(345)) - Shutting down container container_1463523513603_0001_00_000002
2016-05-17 22:18:34,142 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(270)) - ContainerExecutionResult for: container_1463523513603_0001_00_000002 = ContainerExecutionResult{exitStatus=SUCCESS, throwable=null, errorMessage='Asked to die by the AM'}
2016-05-17 22:18:34,143 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(274)) - Container: container_1463523513603_0001_00_000002 completed successfully
2016-05-17 22:18:34,143 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1463523513603_0001_1][Event:DAG_COMMIT_STARTED]: dagID=dag_1463523513603_0001_1
2016-05-17 22:18:34,143 INFO impl.DAGImpl (DAGImpl.java:commitOrAbortOutputs(871)) - No output committers for vertex: vertex_1463523513603_0001_1_00 [AC44F8ABA3AC42CBB216039CD59B227D]
2016-05-17 22:18:34,143 INFO impl.DAGImpl (DAGImpl.java:commitOrAbortOutputs(892)) - Committing output: B6674A59136C4FE092506740C4EA5E16 for vertex: vertex_1463523513603_0001_1_01 [A95EBDDFBC0344098A63024BC7984A3D]
2016-05-17 22:18:34,152 INFO recovery.RecoveryService (RecoveryService.java:handle(306)) - DAG completed, dagId=dag_1463523513603_0001_1, queueSize=0
2016-05-17 22:18:34,153 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event DAG_FINISHED to history file
2016-05-17 22:18:34,153 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1463523513603_0001_1][Event:DAG_FINISHED]: dagId=dag_1463523513603_0001_1, startTime=1463523513777, finishTime=1463523514150, timeTaken=373, status=SUCCEEDED, diagnostics=, counters=Counters: 55, org.apache.tez.common.counters.DAGCounter, NUM_SUCCEEDED_TASKS=2, TOTAL_LAUNCHED_TASKS=2, RACK_LOCAL_TASKS=1, File System Counters, FILE_BYTES_READ=234, FILE_BYTES_WRITTEN=209, FILE_READ_OPS=0, FILE_LARGE_READ_OPS=0, FILE_WRITE_OPS=0, org.apache.tez.common.counters.TaskCounter, REDUCE_INPUT_GROUPS=4, REDUCE_INPUT_RECORDS=5, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=10, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, GC_TIME_MILLIS=183, CPU_MILLISECONDS=0, PHYSICAL_MEMORY_BYTES=0, VIRTUAL_MEMORY_BYTES=0, COMMITTED_HEAP_BYTES=1224736768, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=10, OUTPUT_BYTES=105, OUTPUT_BYTES_WITH_OVERHEAD=121, OUTPUT_BYTES_PHYSICAL=125, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=125, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_BYTES=125, SHUFFLE_BYTES_DECOMPRESSED=121, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=125, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=9, MERGE_PHASE_TIME=12, FIRST_EVENT_RECEIVED=7, LAST_EVENT_RECEIVED=7, Shuffle Errors, BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0, cascading.flow.SliceCounters, Process_Begin_Time=2927047028173, Process_Duration=7, Process_End_Time=2927047028180, Read_Duration=0, Tuples_Read=10, Tuples_Written=10, Write_Duration=1, cascading.flow.StepCounters, Tuples_Read=5, Tuples_Written=5
2016-05-17 22:18:34,153 INFO impl.DAGImpl (DAGImpl.java:finished(1180)) - DAG: dag_1463523513603_0001_1 finished with state: SUCCEEDED
2016-05-17 22:18:34,153 INFO impl.DAGImpl (DAGImpl.java:handle(973)) - dag_1463523513603_0001_1 transitioned from RUNNING to SUCCEEDED due to event DAG_VERTEX_COMPLETED
2016-05-17 22:18:34,154 INFO container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1463523513603_0001_00_000002 transitioned from STOP_REQUESTED to COMPLETED via event C_COMPLETED
2016-05-17 22:18:34,154 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:stop(242)) - Ignoring stop request for containerId: container_1463523513603_0001_00_000002
2016-05-17 22:18:34 Completed Dag: dag_1463523513603_0001_1
2016-05-17 22:18:34,157 INFO app.DAGAppMaster (DAGAppMaster.java:handle(579)) - DAG completed, dagId=dag_1463523513603_0001_1, dagState=SUCCEEDED
2016-05-17 22:18:34,157 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(164)) - Redirecting log file based on addend: dag_1463523513603_0001_1_post
2016-05-17 22:18:34,158 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(179)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-17 22:18:34,158 INFO app.DAGAppMaster (DAGAppMaster.java:handle(625)) - Waiting for next DAG to be submitted.
2016-05-17 22:18:34,161 INFO client.TezClient (TezClient.java:stop(495)) - Shutting down Tez Session, sessionName=(1/1) ...testSimpleBuffer3/simple3, applicationId=application_1463523513603_0001
2016-05-17 22:18:34,164 INFO client.DAGClientHandler (DAGClientHandler.java:shutdownAM(117)) - Received message to shutdown AM
2016-05-17 22:18:34,164 INFO rm.TaskSchedulerEventHandler (TaskSchedulerEventHandler.java:setShouldUnregisterFlag(580)) - TaskScheduler notified that it should unregister from RM
2016-05-17 22:18:34,164 INFO app.DAGAppMaster (DAGAppMaster.java:shutdownTezAM(1088)) - No current running DAG, shutting down the AM
2016-05-17 22:18:34,165 INFO app.DAGAppMaster (DAGAppMaster.java:shutdown(692)) - DAGAppMasterShutdownHandler invoked
2016-05-17 22:18:34,165 INFO app.DAGAppMaster (DAGAppMaster.java:shutdown(701)) - Handling DAGAppMaster shutdown
2016-05-17 22:18:34,165 INFO app.DAGAppMaster (DAGAppMaster.java:run(721)) - Sleeping for 5 seconds before shutting down
2016-05-17 22:18:34,166 INFO history.HistoryEventHandler (HistoryEventHandler.java:serviceStop(79)) - Stopping HistoryEventHandler
2016-05-17 22:18:34,166 INFO recovery.RecoveryService (RecoveryService.java:serviceStop(189)) - Stopping RecoveryService
2016-05-17 22:18:34,166 INFO recovery.RecoveryService (RecoveryService.java:serviceStop(192)) - Handle the remaining events in queue, queue size=0
2016-05-17 22:18:34,166 INFO recovery.RecoveryService (RecoveryService.java:run(166)) - EventQueue take interrupted. Returning
2016-05-17 22:18:34,167 INFO recovery.RecoveryService (RecoveryService.java:serviceStop(214)) - Closing Summary Stream
2016-05-17 22:18:34,167 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:serviceStop(118)) - Stopping SimpleHistoryLoggingService, eventQueueBacklog=0
2016-05-17 22:18:34,167 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:run(105)) - EventQueue take interrupted. Returning
2016-05-17 22:18:34,172 INFO app.DAGAppMaster (DAGAppMaster.java:run(1773)) - Completed deletion of tez scratch data dir, path=file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.5.x-tez-0.6.x/build/test/tmp/.staging/86EF4021943745A2BAAB6F46343D991A/E27B02069D014C1E84A753853F2C4529/.tez/application_1463523513603_0001
2016-05-17 22:18:34,173 INFO util.Hadoop18TapUtil (Hadoop18TapUtil.java:cleanTempPath(226)) - deleting temp path /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.5.x-tez-0.6.x/build/test/output/hadoop2-tez/bufferpipesplatform/testSimpleBuffer3/simple3/_temporary
2016-05-17 22:18:34,173 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [test] completed in: 00:00.590
2016-05-17 22:18:34,248 INFO cascading.PlatformTestCase (PlatformTestCase.java:tearDown(189)) - copying to local /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.5.x-tez-0.6.x/build/test/output//hadoop2-tez/bufferpipesplatform/testSimpleBuffer3/simple3
2016-05-17 22:18:34,275 INFO util.Util (Util.java:findMainClass(1374)) - using default application jar, may cause class not found exceptions on the cluster
2016-05-17 22:18:34,276 INFO planner.Hadoop2TezPlanner (Hadoop2TezPlanner.java:initialize(151)) - using application jar: /opt/concurrent/home/teamcity/.gradle/caches/modules-2/files-2.1/cascading/cascading-hadoop2-tez/3.1.0/226af02e24d5195c60e4ebe911d0c7c9780f59f0/cascading-hadoop2-tez-3.1.0.jar
2016-05-17 22:18:34,318 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [test] executed rule registry: NoHashJoinHadoop2TezRuleRegistry, completed as: SUCCESS, in: 00:00.017
2016-05-17 22:18:34,333 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [test] executed rule registry: HashJoinHadoop2TezRuleRegistry, completed as: SUCCESS, in: 00:00.032
2016-05-17 22:18:34,333 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [test] rule registry: NoHashJoinHadoop2TezRuleRegistry, supports assembly with steps: 1, nodes: 2
2016-05-17 22:18:34,333 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [test] rule registry: HashJoinHadoop2TezRuleRegistry, supports assembly with steps: 1, nodes: 2
2016-05-17 22:18:34,334 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [test] rule registry: NoHashJoinHadoop2TezRuleRegistry, result was selected using: 'default comparator: selects plan with fewest steps and fewest nodes'
2016-05-17 22:18:34,349 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [test] tap forcing step to tez local mode: /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.5.x-tez-0.6.x/build/classes/test/data/lhs.txt
2016-05-17 22:18:34,355 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [test] tap forcing step to tez local mode: /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.5.x-tez-0.6.x/build/test/output/hadoop2-tez/bufferpipesplatform/testFirstNBufferReverse/firstn
2016-05-17 22:18:34,403 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [test] starting
2016-05-17 22:18:34,404 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [test] source: Hfs["TextDelimited[['num', 'lower']]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.5.x-tez-0.6.x/build/classes/test/data/lhs.txt"]
2016-05-17 22:18:34,404 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [test] sink: Hfs["TextDelimited[['num', 'lower']]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.5.x-tez-0.6.x/build/test/output/hadoop2-tez/bufferpipesplatform/testFirstNBufferReverse/firstn"]
2016-05-17 22:18:34,404 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [test] parallel execution of steps is enabled: false
2016-05-17 22:18:34,404 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [test] executing total steps: 1
2016-05-17 22:18:34,404 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [test] allocating management threads: 1
2016-05-17 22:18:34,405 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [test] starting step: (1/1) ...irstNBufferReverse/firstn
2016-05-17 22:18:34,405 WARN flow.Flow (BaseFlow.java:logWarn(1516)) - [test] 'yarn.timeline-service.enabled' is disabled, please enable to capture detailed metrics of completed flows, this may require starting the YARN timeline server daemon
2016-05-17 22:18:34,417 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [test] tez session mode enabled: true
2016-05-17 22:18:34,422 INFO client.TezClient (TezClient.java:<init>(153)) - Tez Client Version: [ component=tez-api, version=0.6.2, revision=c04283bf91fd96823eec37056d5c8b1a92728265, SCM-URL=scm:git:https://git-wip-us.apache.org/repos/asf/tez.git, buildTime=20150804-1436 ]
2016-05-17 22:18:34,422 INFO client.TezClient (TezClient.java:start(333)) - Session mode. Starting session.
2016-05-17 22:18:34,423 INFO client.TezClientUtils (TezClientUtils.java:setupTezJarsLocalResources(159)) - Ignoring 'tez.lib.uris' since 'tez.ignore.lib.uris' is set to true
2016-05-17 22:18:34,432 INFO client.TezClient (TezCommonUtils.java:createTezSystemStagingPath(118)) - Tez system stage directory file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.5.x-tez-0.6.x/build/test/tmp/.staging/C886F761D5D24A6BA33C0106FE67C18A/DB6F7C03C9E6433D8612EA31AE545BC1/.tez/application_1463523514422_0001 doesn't exist and is created
2016-05-17 22:18:34,449 INFO client.LocalClient (LocalClient.java:createDAGAppMaster(328)) - DAGAppMaster thread has been created
2016-05-17 22:18:34,449 INFO client.LocalClient (LocalClient.java:startDAGAppMaster(251)) - DAGAppMaster is not created wait for 100ms...
2016-05-17 22:18:34,449 INFO client.LocalClient (LocalClient.java:run(275)) - Using staging directory: /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.5.x-tez-0.6.x/build/test/tmp/.staging/C886F761D5D24A6BA33C0106FE67C18A/DB6F7C03C9E6433D8612EA31AE545BC1/.tez/application_1463523514422_0001
2016-05-17 22:18:34,453 INFO app.DAGAppMaster (DAGAppMaster.java:<init>(309)) - Created DAGAppMaster for application appattempt_1463523514422_0001_000000, versionInfo=[ component=tez-dag, version=0.6.2, revision=c04283bf91fd96823eec37056d5c8b1a92728265, SCM-URL=scm:git:https://git-wip-us.apache.org/repos/asf/tez.git, buildTime=20150804-1436 ]
2016-05-17 22:18:34,464 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(333)) - Comparing client version with AM version, clientVersion=0.6.2, AMVersion=0.6.2
2016-05-17 22:18:34,465 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(386)) - Adding session token to jobTokenSecretManager for application
2016-05-17 22:18:34,466 INFO node.AMNodeTracker (AMNodeTracker.java:serviceInit(77)) - blacklistDisablePercent is 33, blacklistingEnabled: false, maxTaskFailuresPerNode: 1
2016-05-17 22:18:34,466 INFO history.HistoryEventHandler (HistoryEventHandler.java:serviceInit(50)) - Initializing HistoryEventHandler
2016-05-17 22:18:34,467 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:serviceInit(66)) - Log file location for SimpleHistoryLoggingService not specified, defaulting to containerLogDir=/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.5.x-tez-0.6.x/build/test/tmp/.staging/C886F761D5D24A6BA33C0106FE67C18A/DB6F7C03C9E6433D8612EA31AE545BC1/.tez/application_1463523514422_0001/localmode-log-dir
2016-05-17 22:18:34,467 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:serviceInit(88)) - Initializing SimpleHistoryLoggingService, logFileLocation=/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.5.x-tez-0.6.x/build/test/tmp/.staging/C886F761D5D24A6BA33C0106FE67C18A/DB6F7C03C9E6433D8612EA31AE545BC1/.tez/application_1463523514422_0001/localmode-log-dir/history.txt.appattempt_1463523514422_0001_000000, maxErrors=10
2016-05-17 22:18:34,467 INFO recovery.RecoveryService (RecoveryService.java:serviceInit(109)) - Initializing RecoveryService
2016-05-17 22:18:34,467 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:N/A][Event:AM_LAUNCHED]: appAttemptId=appattempt_1463523514422_0001_000000, appSubmitTime=1463523514453, launchTime=1463523514453
2016-05-17 22:18:34,471 INFO history.HistoryEventHandler (HistoryEventHandler.java:serviceStart(73)) - Starting HistoryEventHandler
2016-05-17 22:18:34,474 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:serviceStart(95)) - Starting SimpleHistoryLoggingService
2016-05-17 22:18:34,475 INFO client.DAGClientServer (DAGClientServer.java:serviceStart(91)) - Instantiated DAGClientRPCServer at ip-10-123-148-208.ec2.internal/10.123.148.208:51070
2016-05-17 22:18:34,477 INFO recovery.RecoveryService (RecoveryService.java:serviceStart(127)) - Starting RecoveryService
2016-05-17 22:18:34,477 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event AM_LAUNCHED to history file
2016-05-17 22:18:34,478 WARN impl.MetricsConfig (MetricsConfig.java:loadFirst(124)) - Cannot locate configuration: tried hadoop-metrics2-dagappmaster.properties,hadoop-metrics2.properties
2016-05-17 22:18:34,480 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:N/A][Event:AM_STARTED]: appAttemptId=appattempt_1463523514422_0001_000000, startTime=1463523514480
2016-05-17 22:18:34,480 INFO app.DAGAppMaster (DAGAppMaster.java:serviceStart(1644)) - In Session mode. Waiting for DAG over RPC
2016-05-17 22:18:34,480 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event AM_STARTED to history file
2016-05-17 22:18:34,549 INFO client.LocalClient (LocalClient.java:startDAGAppMaster(236)) - DAGAppMaster state: IDLE
2016-05-17 22:18:34,549 INFO client.TezClient (TezClient.java:start(367)) - The url to track the Tez Session: N/A
2016-05-17 22:18:34,549 INFO client.TezClient (TezClient.java:submitDAGSession(405)) - Submitting dag to TezSession, sessionName=(1/1) ...irstNBufferReverse/firstn, applicationId=application_1463523514422_0001, dagName=[C886F761D5D24A6BA33C0106FE67C18A/DB6F7C03C9E6433D8612EA31AE545BC1] test/(1/1) ...irstNBufferReverse/firstn
2016-05-17 22:18:34,567 INFO app.DAGAppMaster (DAGAppMaster.java:submitDAGToAppMaster(1126)) - Starting DAG submitted via RPC: [C886F761D5D24A6BA33C0106FE67C18A/DB6F7C03C9E6433D8612EA31AE545BC1] test/(1/1) ...irstNBufferReverse/firstn
2016-05-17 22:18:34,568 INFO app.DAGAppMaster (DAGAppMaster.java:generateDAGVizFile(870)) - Generating DAG graphviz file, dagId=dag_1463523514422_0001_1, filePath=/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.5.x-tez-0.6.x/build/test/tmp/.staging/C886F761D5D24A6BA33C0106FE67C18A/DB6F7C03C9E6433D8612EA31AE545BC1/.tez/application_1463523514422_0001/localmode-log-dir/dag_1463523514422_0001_1.dot
2016-05-17 22:18:34,569 INFO app.DAGAppMaster (DAGAppMaster.java:writePBTextFile(887)) - Writing DAG plan to: /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.5.x-tez-0.6.x/build/test/tmp/.staging/C886F761D5D24A6BA33C0106FE67C18A/DB6F7C03C9E6433D8612EA31AE545BC1/.tez/application_1463523514422_0001/localmode-log-dir/dag_1463523514422_0001_1-tez-dag.pb.txt
2016-05-17 22:18:34,579 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(164)) - Redirecting log file based on addend: dag_1463523514422_0001_1
2016-05-17 22:18:34,579 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(179)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-17 22:18:34,579 INFO app.DAGAppMaster (DAGAppMaster.java:startDAG(2051)) - Running DAG: [C886F761D5D24A6BA33C0106FE67C18A/DB6F7C03C9E6433D8612EA31AE545BC1] test/(1/1) ...irstNBufferReverse/firstn
2016-05-17 22:18:34 Running Dag: dag_1463523514422_0001_1
2016-05-17 22:18:34,593 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event DAG_SUBMITTED to history file
2016-05-17 22:18:34,593 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1463523514422_0001_1][Event:DAG_SUBMITTED]: dagID=dag_1463523514422_0001_1, submitTime=1463523514567
2016-05-17 22:18:34,594 INFO impl.VertexImpl (VertexImpl.java:setAdditionalOutputs(4033)) - setting additional outputs for vertex F7D0FABC2C6A4B3B909C555D58A16B58
2016-05-17 22:18:34,594 INFO impl.DAGImpl (DAGImpl.java:assignDAGScheduler(1381)) - Using DAG Scheduler: org.apache.tez.dag.app.dag.impl.DAGSchedulerNaturalOrderControlled
2016-05-17 22:18:34,595 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event DAG_INITIALIZED to history file
2016-05-17 22:18:34,595 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1463523514422_0001_1][Event:DAG_INITIALIZED]: dagID=dag_1463523514422_0001_1, initTime=1463523514594
2016-05-17 22:18:34,595 INFO impl.DAGImpl (DAGImpl.java:handle(973)) - dag_1463523514422_0001_1 transitioned from NEW to INITED due to event DAG_INIT
2016-05-17 22:18:34,597 INFO client.TezClient (TezClient.java:submitDAGSession(478)) - Submitted dag to TezSession, sessionName=(1/1) ...irstNBufferReverse/firstn, applicationId=application_1463523514422_0001, dagName=[C886F761D5D24A6BA33C0106FE67C18A/DB6F7C03C9E6433D8612EA31AE545BC1] test/(1/1) ...irstNBufferReverse/firstn
2016-05-17 22:18:34,598 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1463523514422_0001_1][Event:DAG_STARTED]: dagID=dag_1463523514422_0001_1, startTime=1463523514596
2016-05-17 22:18:34,598 INFO impl.DAGImpl (DAGImpl.java:transition(1614)) - Added additional resources : [[]] to classpath
2016-05-17 22:18:34,598 INFO impl.DAGImpl (DAGImpl.java:handle(973)) - dag_1463523514422_0001_1 transitioned from INITED to RUNNING due to event DAG_START
2016-05-17 22:18:34,598 INFO impl.VertexImpl (VertexImpl.java:setupVertex(2068)) - Root Inputs exist for Vertex: C982EAFB40CC492D8D2F1992C1E8AEF9 : {9C35570ADFCA474AAF1AE528C937CC58={InputName=9C35570ADFCA474AAF1AE528C937CC58}, {Descriptor=ClassName=org.apache.tez.mapreduce.input.MRInput, hasPayload=true}, {ControllerDescriptor=ClassName=org.apache.tez.mapreduce.common.MRInputAMSplitGenerator, hasPayload=false}}
2016-05-17 22:18:34,599 INFO impl.VertexImpl (VertexImpl.java:setupVertex(2078)) - Starting root input initializer for input: 9C35570ADFCA474AAF1AE528C937CC58, with class: [org.apache.tez.mapreduce.common.MRInputAMSplitGenerator]
2016-05-17 22:18:34,599 INFO impl.VertexImpl (VertexImpl.java:assignVertexManager(2191)) - Setting vertexManager to RootInputVertexManager for vertex_1463523514422_0001_1_00 [C982EAFB40CC492D8D2F1992C1E8AEF9]
2016-05-17 22:18:34,599 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(2983)) - Num tasks is -1. Expecting VertexManager/InputInitializers/1-1 split to set #tasks for the vertex vertex_1463523514422_0001_1_00 [C982EAFB40CC492D8D2F1992C1E8AEF9]
2016-05-17 22:18:34,599 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(2987)) - Vertex will initialize from input initializer. vertex_1463523514422_0001_1_00 [C982EAFB40CC492D8D2F1992C1E8AEF9]
2016-05-17 22:18:34,599 INFO impl.VertexImpl (VertexImpl.java:setupInputInitializerManager(3933)) - Vertex will initialize via inputInitializers vertex_1463523514422_0001_1_00 [C982EAFB40CC492D8D2F1992C1E8AEF9]. Starting root input initializers: 1
2016-05-17 22:18:34,605 INFO impl.VertexImpl (VertexImpl.java:handle(1564)) - vertex_1463523514422_0001_1_00 [C982EAFB40CC492D8D2F1992C1E8AEF9] transitioned from NEW to INITIALIZING due to event V_INIT
2016-05-17 22:18:34,605 INFO impl.VertexImpl (VertexImpl.java:assignVertexManager(2203)) - Setting vertexManager to ShuffleVertexManager for vertex_1463523514422_0001_1_01 [F7D0FABC2C6A4B3B909C555D58A16B58]
2016-05-17 22:18:34,605 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event DAG_STARTED to history file
2016-05-17 22:18:34,608 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:initialize(696)) - Shuffle Vertex Manager: settings minFrac:0.25 maxFrac:0.75 auto:false desiredTaskIput:104857600 minTasks:1
2016-05-17 22:18:34,610 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(3014)) - Creating 1 tasks for vertex: vertex_1463523514422_0001_1_01 [F7D0FABC2C6A4B3B909C555D58A16B58]
2016-05-17 22:18:34,610 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(3026)) - Directly initializing vertex: vertex_1463523514422_0001_1_01 [F7D0FABC2C6A4B3B909C555D58A16B58]
2016-05-17 22:18:34,610 INFO impl.VertexImpl (VertexImpl.java:initializeCommitters(1915)) - Invoking committer inits for vertex, vertexId=vertex_1463523514422_0001_1_01 [F7D0FABC2C6A4B3B909C555D58A16B58]
2016-05-17 22:18:34,609 INFO dag.RootInputInitializerManager (RootInputInitializerManager.java:run(242)) - Starting InputInitializer for Input: 9C35570ADFCA474AAF1AE528C937CC58 on vertex vertex_1463523514422_0001_1_00 [C982EAFB40CC492D8D2F1992C1E8AEF9]
2016-05-17 22:18:34,612 INFO common.MRInputAMSplitGenerator (MRInputAMSplitGenerator.java:initialize(92)) - Emitting serialized splits: true
2016-05-17 22:18:34,612 INFO common.MRInputAMSplitGenerator (MRInputAMSplitGenerator.java:initialize(110)) - Input 9C35570ADFCA474AAF1AE528C937CC58 asking for 1 tasks. Headroom: 672 Task Resource: 1024 waves: 1.7
2016-05-17 22:18:34,614 INFO impl.VertexImpl (VertexImpl.java:initializeCommitters(1927)) - Instantiating committer for output=8C05B316B49B4068837D5FC66B1D2E75, vertexId=vertex_1463523514422_0001_1_01 [F7D0FABC2C6A4B3B909C555D58A16B58], committerClass=org.apache.tez.mapreduce.committer.MROutputCommitter
2016-05-17 22:18:34,614 INFO common.MRInputAMSplitGenerator (MRInputAMSplitGenerator.java:initialize(121)) - Grouping input splits
2016-05-17 22:18:34,614 INFO impl.VertexImpl (VertexImpl.java:run(1945)) - Invoking committer init for output=8C05B316B49B4068837D5FC66B1D2E75, vertexId=vertex_1463523514422_0001_1_01 [F7D0FABC2C6A4B3B909C555D58A16B58]
2016-05-17 22:18:34,614 INFO hadoop.MRInputHelpers (MRInputHelpers.java:generateInputSplitsToMem(294)) - Generating mapred api input splits
2016-05-17 22:18:34,616 INFO committer.MROutputCommitter (MROutputCommitter.java:getOutputCommitter(145)) - OutputCommitter for outputName=8C05B316B49B4068837D5FC66B1D2E75, vertexName=F7D0FABC2C6A4B3B909C555D58A16B58, outputCommitterClass=org.apache.hadoop.mapred.FileOutputCommitter
2016-05-17 22:18:34,617 INFO impl.VertexImpl (VertexImpl.java:run(1949)) - Invoking committer setup for output=8C05B316B49B4068837D5FC66B1D2E75, vertexId=vertex_1463523514422_0001_1_01 [F7D0FABC2C6A4B3B909C555D58A16B58]
2016-05-17 22:18:34,619 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [test] submitted tez dag to app master: application_1463523514422_0001, with dag id: dag_1463523514422_0001_1
2016-05-17 22:18:34,616 INFO hadoop.MRInputHelpers (MRInputHelpers.java:generateInputSplitsToMem(300)) - NumSplits: 1, SerializedSize: 396
2016-05-17 22:18:34,620 INFO dag.RootInputInitializerManager (RootInputInitializerManager.java:onSuccess(272)) - Succeeded InputInitializer for Input: 9C35570ADFCA474AAF1AE528C937CC58 on vertex vertex_1463523514422_0001_1_00 [C982EAFB40CC492D8D2F1992C1E8AEF9]
2016-05-17 22:18:34,621 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event VERTEX_INITIALIZED to history file
2016-05-17 22:18:34,621 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1463523514422_0001_1][Event:VERTEX_INITIALIZED]: vertexName=F7D0FABC2C6A4B3B909C555D58A16B58, vertexId=vertex_1463523514422_0001_1_01, initRequestedTime=1463523514605, initedTime=1463523514621, numTasks=1, processorName=cascading.flow.tez.FlowProcessor, additionalInputsCount=0
2016-05-17 22:18:34,622 INFO impl.VertexImpl (VertexImpl.java:handle(1564)) - vertex_1463523514422_0001_1_01 [F7D0FABC2C6A4B3B909C555D58A16B58] transitioned from NEW to INITED due to event V_INIT
2016-05-17 22:18:34,623 INFO impl.VertexImpl (VertexImpl.java:canInitVertex(3195)) - Cannot init vertex: vertex_1463523514422_0001_1_00 [C982EAFB40CC492D8D2F1992C1E8AEF9] numTasks: -1 numUnitializedEdges: 0 numInitializedInputs: 0 initWaitsForRootInitializers: true
2016-05-17 22:18:34,623 INFO impl.VertexImpl (VertexImpl.java:setParallelism(1374)) - Got updated RootInputsSpecs: {9C35570ADFCA474AAF1AE528C937CC58=forAllWorkUnits=true, update=[1]}
2016-05-17 22:18:34,624 INFO impl.VertexImpl (VertexImpl.java:setParallelism(1393)) - Vertex vertex_1463523514422_0001_1_00 [C982EAFB40CC492D8D2F1992C1E8AEF9] parallelism set to 1
2016-05-17 22:18:34,624 INFO impl.VertexImpl (VertexImpl.java:canInitVertex(3195)) - Cannot init vertex: vertex_1463523514422_0001_1_00 [C982EAFB40CC492D8D2F1992C1E8AEF9] numTasks: 1 numUnitializedEdges: 0 numInitializedInputs: 0 initWaitsForRootInitializers: true
2016-05-17 22:18:34,624 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1463523514422_0001_1][Event:VERTEX_DATA_MOVEMENT_EVENTS_GENERATED]: vertexId=vertex_1463523514422_0001_1_00, eventCount=1
2016-05-17 22:18:34,632 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1463523514422_0001_1][Event:VERTEX_INITIALIZED]: vertexName=C982EAFB40CC492D8D2F1992C1E8AEF9, vertexId=vertex_1463523514422_0001_1_00, initRequestedTime=1463523514598, initedTime=1463523514632, numTasks=1, processorName=cascading.flow.tez.FlowProcessor, additionalInputsCount=1
2016-05-17 22:18:34,633 INFO impl.VertexImpl (VertexImpl.java:startIfPossible(3061)) - Triggering start event for vertex: vertex_1463523514422_0001_1_00 [C982EAFB40CC492D8D2F1992C1E8AEF9] with distanceFromRoot: 0
2016-05-17 22:18:34,633 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event VERTEX_INITIALIZED to history file
2016-05-17 22:18:34,633 INFO impl.VertexImpl (VertexImpl.java:handle(1564)) - vertex_1463523514422_0001_1_00 [C982EAFB40CC492D8D2F1992C1E8AEF9] transitioned from INITIALIZING to INITED due to event V_ROOT_INPUT_INITIALIZED
2016-05-17 22:18:34,633 INFO impl.ImmediateStartVertexManager (ImmediateStartVertexManager.java:scheduleTasks(99)) - Starting 1 in C982EAFB40CC492D8D2F1992C1E8AEF9
2016-05-17 22:18:34,634 INFO impl.VertexImpl (VertexImpl.java:scheduleTasks(1250)) - Routing pending task events for vertex: vertex_1463523514422_0001_1_00 [C982EAFB40CC492D8D2F1992C1E8AEF9]
2016-05-17 22:18:34,634 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event VERTEX_STARTED to history file
2016-05-17 22:18:34,634 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1463523514422_0001_1][Event:VERTEX_STARTED]: vertexId=vertex_1463523514422_0001_1_00, startRequestedTime=1463523514633, startedTime=1463523514633
2016-05-17 22:18:34,635 INFO impl.VertexImpl (VertexImpl.java:handle(1564)) - vertex_1463523514422_0001_1_00 [C982EAFB40CC492D8D2F1992C1E8AEF9] transitioned from INITED to RUNNING due to event V_START
2016-05-17 22:18:34,640 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event TASK_STARTED to history file
2016-05-17 22:18:34,640 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1463523514422_0001_1][Event:TASK_STARTED]: vertexName=C982EAFB40CC492D8D2F1992C1E8AEF9, taskId=task_1463523514422_0001_1_00_000000, scheduledTime=1463523514640, launchTime=1463523514640
2016-05-17 22:18:34,641 INFO impl.TaskImpl (TaskImpl.java:handle(862)) - task_1463523514422_0001_1_00_000000 Task Transitioned from NEW to SCHEDULED due to event T_SCHEDULE
2016-05-17 22:18:34,641 INFO impl.VertexImpl (VertexImpl.java:transition(3155)) - Source vertex started: vertex_1463523514422_0001_1_00 for vertex: vertex_1463523514422_0001_1_01 [F7D0FABC2C6A4B3B909C555D58A16B58] numStartedSources: 1 numSources: 1
2016-05-17 22:18:34,641 INFO impl.VertexImpl (VertexImpl.java:startIfPossible(3061)) - Triggering start event for vertex: vertex_1463523514422_0001_1_01 [F7D0FABC2C6A4B3B909C555D58A16B58] with distanceFromRoot: 1
2016-05-17 22:18:34,642 INFO impl.DAGSchedulerNaturalOrderControlled (DAGSchedulerNaturalOrderControlled.java:scheduleTask(111)) - Scheduled vertex: vertex_1463523514422_0001_1_00 [C982EAFB40CC492D8D2F1992C1E8AEF9]
2016-05-17 22:18:34,642 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:handleVertexStateUpdate(720)) - Received configured notification : CONFIGURED for vertex: C982EAFB40CC492D8D2F1992C1E8AEF9 in vertex: F7D0FABC2C6A4B3B909C555D58A16B58
2016-05-17 22:18:34,643 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:onVertexStarted(355)) - OnVertexStarted vertex: F7D0FABC2C6A4B3B909C555D58A16B58 with 1 source tasks and 1 pending tasks
2016-05-17 22:18:34,644 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event VERTEX_STARTED to history file
2016-05-17 22:18:34,644 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1463523514422_0001_1][Event:VERTEX_STARTED]: vertexId=vertex_1463523514422_0001_1_01, startRequestedTime=1463523514642, startedTime=1463523514642
2016-05-17 22:18:34,644 INFO impl.VertexImpl (VertexImpl.java:handle(1564)) - vertex_1463523514422_0001_1_01 [F7D0FABC2C6A4B3B909C555D58A16B58] transitioned from INITED to RUNNING due to event V_START
2016-05-17 22:18:34,645 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:transition(1014)) - remoteTaskSpec:DAGName : [C886F761D5D24A6BA33C0106FE67C18A/DB6F7C03C9E6433D8612EA31AE545BC1] test/(1/1) ...irstNBufferReverse/firstn, VertexName: C982EAFB40CC492D8D2F1992C1E8AEF9, VertexParallelism: 1, TaskAttemptID:attempt_1463523514422_0001_1_00_000000_0, processorName=cascading.flow.tez.FlowProcessor, inputSpecListSize=1, outputSpecListSize=1, inputSpecList=[{{ sourceVertexName=9C35570ADFCA474AAF1AE528C937CC58, physicalEdgeCount=1, inputClassName=org.apache.tez.mapreduce.input.MRInput }}, ], outputSpecList=[{{ destinationVertexName=F7D0FABC2C6A4B3B909C555D58A16B58, physicalEdgeCount=1, outputClassName=org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput }}, ]
2016-05-17 22:18:34,645 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:handle(692)) - attempt_1463523514422_0001_1_00_000000_0 TaskAttempt Transitioned from NEW to START_WAIT due to event TA_SCHEDULE
2016-05-17 22:18:34,646 INFO rm.TaskSchedulerEventHandler (TaskSchedulerEventHandler.java:handleEvent(149)) - Processing the event EventType: S_TA_LAUNCH_REQUEST
2016-05-17 22:18:34,647 INFO node.AMNodeTracker (AMNodeTracker.java:nodeSeen(91)) - Adding new node: 127.0.0.1:0
2016-05-17 22:18:34,647 INFO container.AMContainerHelpers (AMContainerHelpers.java:createCommonContainerLaunchContext(112)) - Adding #1 tokens and #0 secret keys for NM use for launching container
2016-05-17 22:18:34,647 INFO container.AMContainerHelpers (AMContainerHelpers.java:createCommonContainerLaunchContext(122)) - Putting shuffle token in serviceData
2016-05-17 22:18:34,648 INFO container.AMContainerImpl (AMContainerImpl.java:transition(521)) - Sending Launch Request for Container with id: container_1463523514422_0001_00_000001
2016-05-17 22:18:34,648 INFO container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1463523514422_0001_00_000001 transitioned from ALLOCATED to LAUNCHING via event C_LAUNCH_REQUEST
2016-05-17 22:18:34,649 INFO task.TezChild (TezChild.java:newTezChild(432)) - PID, containerIdentifier: null, container_1463523514422_0001_00_000001
2016-05-17 22:18:34,649 WARN impl.MetricsSystemImpl (MetricsSystemImpl.java:init(145)) - DAGAppMaster metrics system already initialized!
2016-05-17 22:18:34,650 INFO container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1463523514422_0001_00_000001 transitioned from LAUNCHING to IDLE via event C_LAUNCHED
2016-05-17 22:18:34,650 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1463523514422_0001_1][Event:CONTAINER_LAUNCHED]: containerId=container_1463523514422_0001_00_000001, launchTime=1463523514650
2016-05-17 22:18:34,650 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event CONTAINER_LAUNCHED to history file
2016-05-17 22:18:34,652 INFO task.ContainerReporter (ContainerReporter.java:call(55)) - Attempting to fetch new task for container container_1463523514422_0001_00_000001
2016-05-17 22:18:34,653 INFO container.AMContainerImpl (AMContainerImpl.java:transition(827)) - Assigned taskAttempt + [attempt_1463523514422_0001_1_00_000000_0] to container: [container_1463523514422_0001_00_000001]
2016-05-17 22:18:34,653 INFO container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1463523514422_0001_00_000001 transitioned from IDLE to RUNNING via event C_PULL_TA
2016-05-17 22:18:34,653 INFO app.TaskAttemptListenerImpTezDag (TaskAttemptListenerImpTezDag.java:getTask(240)) - Container with id: container_1463523514422_0001_00_000001 given task: attempt_1463523514422_0001_1_00_000000_0
2016-05-17 22:18:34,653 INFO task.ContainerReporter (ContainerReporter.java:call(65)) - Got TaskUpdate for containerId= container_1463523514422_0001_00_000001: 0 ms after starting to poll. TaskInfo: shouldDie: false, currentTaskAttemptId: attempt_1463523514422_0001_1_00_000000_0
2016-05-17 22:18:34,653 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:transition(1146)) - TaskAttempt: [attempt_1463523514422_0001_1_00_000000_0] started. Is using containerId: [container_1463523514422_0001_00_000001] on NM: [127.0.0.1:0]
2016-05-17 22:18:34,654 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event TASK_ATTEMPT_STARTED to history file
2016-05-17 22:18:34 Starting to run new task attempt: attempt_1463523514422_0001_1_00_000000_0
2016-05-17 22:18:34,654 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(164)) - Redirecting log file based on addend: attempt_1463523514422_0001_1_00_000000_0
2016-05-17 22:18:34,654 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1463523514422_0001_1][Event:TASK_ATTEMPT_STARTED]: vertexName=C982EAFB40CC492D8D2F1992C1E8AEF9, taskAttemptId=attempt_1463523514422_0001_1_00_000000_0, startTime=1463523514653, containerId=container_1463523514422_0001_00_000001, nodeId=127.0.0.1:0, inProgressLogs=127.0.0.1:0/node/containerlogs/container_1463523514422_0001_00_000001/teamcity, completedLogs=
2016-05-17 22:18:34,655 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(179)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-17 22:18:34,655 INFO task.TezChild (TezChild.java:handleNewTaskCredentials(273)) - Refreshing UGI since Credentials have changed
2016-05-17 22:18:34,655 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:handle(692)) - attempt_1463523514422_0001_1_00_000000_0 TaskAttempt Transitioned from START_WAIT to RUNNING due to event TA_STARTED_REMOTELY
2016-05-17 22:18:34,655 INFO impl.TaskImpl (TaskImpl.java:handle(862)) - task_1463523514422_0001_1_00_000000 Task Transitioned from SCHEDULED to RUNNING due to event T_ATTEMPT_LAUNCHED
2016-05-17 22:18:34,655 INFO task.TezChild (TezChild.java:handleNewTaskCredentials(276)) - Credentials : #Tokens=1, #SecretKeys=0
2016-05-17 22:18:34,656 INFO task.TezChild (TezChild.java:handleNewTaskLocalResources(302)) - Localizing additional local resources for Task : {}
2016-05-17 22:18:34,656 INFO task.TezChild (TezChild.java:handleNewTaskLocalResources(312)) - Done localizing additional resources
2016-05-17 22:18:34,656 INFO metrics.TaskCounterUpdater (TaskCounterUpdater.java:initResourceCalculatorPlugin(148)) - Using ResourceCalculatorProcessTree : [ ]
2016-05-17 22:18:34,656 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:<init>(149)) - Initializing LogicalIOProcessorRuntimeTask with TaskSpec: DAGName : [C886F761D5D24A6BA33C0106FE67C18A/DB6F7C03C9E6433D8612EA31AE545BC1] test/(1/1) ...irstNBufferReverse/firstn, VertexName: C982EAFB40CC492D8D2F1992C1E8AEF9, VertexParallelism: 1, TaskAttemptID:attempt_1463523514422_0001_1_00_000000_0, processorName=cascading.flow.tez.FlowProcessor, inputSpecListSize=1, outputSpecListSize=1, inputSpecList=[{{ sourceVertexName=9C35570ADFCA474AAF1AE528C937CC58, physicalEdgeCount=1, inputClassName=org.apache.tez.mapreduce.input.MRInput }}, ], outputSpecList=[{{ destinationVertexName=F7D0FABC2C6A4B3B909C555D58A16B58, physicalEdgeCount=1, outputClassName=org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput }}, ]
2016-05-17 22:18:34,657 INFO resources.MemoryDistributor (MemoryDistributor.java:<init>(85)) - InitialMemoryDistributor (isEnabled=true) invoked with: numInputs=1, numOutputs=1, JVM.maxFree=704643072
2016-05-17 22:18:34,658 INFO task.TezTaskRunner (TezTaskRunner.java:run(172)) - Initializing task, taskAttemptId=attempt_1463523514422_0001_1_00_000000_0
2016-05-17 22:18:34,658 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(191)) - Initializing LogicalProcessorIORuntimeTask
2016-05-17 22:18:34,658 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(195)) - Creating processor, processorClassName=cascading.flow.tez.FlowProcessor
2016-05-17 22:18:34,659 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(389)) - Initializing Input using InputSpec: { sourceVertexName=9C35570ADFCA474AAF1AE528C937CC58, physicalEdgeCount=1, inputClassName=org.apache.tez.mapreduce.input.MRInput }
2016-05-17 22:18:34,660 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:createInput(537)) - Creating Input
2016-05-17 22:18:34,660 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(397)) - Initializing Input with src edge: 9C35570ADFCA474AAF1AE528C937CC58
2016-05-17 22:18:34,663 INFO resources.MemoryDistributor$RequestorInfo (MemoryDistributor.java:<init>(233)) - Received request: 0, type: INPUT, componentVertexName: 9C35570ADFCA474AAF1AE528C937CC58
2016-05-17 22:18:34,661 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(438)) - Initializing Output using OutputSpec: { destinationVertexName=F7D0FABC2C6A4B3B909C555D58A16B58, physicalEdgeCount=1, outputClassName=org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput }
2016-05-17 22:18:34,661 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initializeLogicalIOProcessor(488)) - Initializing processor, processorClassName=cascading.flow.tez.FlowProcessor
2016-05-17 22:18:34,664 INFO impl.TaskImpl (TaskImpl.java:getTaskAttemptTezEvents(481)) - TaskAttempt:attempt_1463523514422_0001_1_00_000000_0 sent events: (0-1)
2016-05-17 22:18:34,664 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:createOutput(565)) - Creating Output
2016-05-17 22:18:34,665 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(446)) - Initializing Output with dest edge: F7D0FABC2C6A4B3B909C555D58A16B58
2016-05-17 22:18:34,667 INFO impl.ExternalSorter (ExternalSorter.java:getInitialMemoryRequirement(293)) - Requested SortBufferSize (tez.runtime.io.sort.mb): 100
2016-05-17 22:18:34,667 INFO tez.FlowProcessor (FlowProcessor.java:initialize(88)) - cascading version: 3.1.0
2016-05-17 22:18:34,667 INFO resources.MemoryDistributor$RequestorInfo (MemoryDistributor.java:<init>(233)) - Received request: 104857600, type: OUTPUT, componentVertexName: F7D0FABC2C6A4B3B909C555D58A16B58
2016-05-17 22:18:34,669 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(452)) - Initialized Output with dest edge: F7D0FABC2C6A4B3B909C555D58A16B58
2016-05-17 22:18:34,670 INFO input.MRInput (MRInput.java:initialize(440)) - Using New mapreduce API: false, split information via event: true
2016-05-17 22:18:34,670 INFO input.MRInput (MRInput.java:initializeInternal(493)) - Initialzed MRInput: 9C35570ADFCA474AAF1AE528C937CC58
2016-05-17 22:18:34,670 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(403)) - Initialized Input with src edge: 9C35570ADFCA474AAF1AE528C937CC58
2016-05-17 22:18:34,677 INFO tez.FlowProcessor (FlowProcessor.java:initialize(94)) - flow node id: C982EAFB40CC492D8D2F1992C1E8AEF9, ordinal: 0
2016-05-17 22:18:34,677 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: C982EAFB40CC492D8D2F1992C1E8AEF9, mem on start (mb), free: 363, total: 584, max: 672
2016-05-17 22:18:34,677 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initializeLogicalIOProcessor(491)) - Initialized processor, processorClassName=cascading.flow.tez.FlowProcessor
2016-05-17 22:18:34,677 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(220)) - Waiting for 2 initializers to finish
2016-05-17 22:18:34,677 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(220)) - Waiting for 1 initializers to finish
2016-05-17 22:18:34,677 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(233)) - All initializers finished
2016-05-17 22:18:34,678 INFO resources.MemoryDistributor (MemoryDistributor.java:makeInitialAllocations(124)) - Using Allocator class: org.apache.tez.runtime.library.resources.WeightedScalingMemoryDistributor
2016-05-17 22:18:34,678 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:getRequestTypeForClass(197)) - Falling back to RequestType.OTHER for class: org.apache.tez.mapreduce.input.MRInput
2016-05-17 22:18:34,678 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:initialProcessMemoryRequestContext(171)) - ScaleFactor: 1, for type: OTHER
2016-05-17 22:18:34,678 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:initialProcessMemoryRequestContext(171)) - ScaleFactor: 12, for type: SORTED_OUTPUT
2016-05-17 22:18:34,678 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:computeReservedFraction(262)) - InitialReservationFraction=0.3, AdditionalReservationFractionForIOs=0.03, finalReserveFractionUsed=0.32999999999999996
2016-05-17 22:18:34,679 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:assignMemory(126)) - Scaling Requests. NumRequests: 2, numScaledRequests: 13, TotalRequested: 104857600, TotalRequestedScaled: 9.679163076923077E7, TotalJVMHeap: 704643072, TotalAvailable: 472110858, TotalRequested/TotalJVMHeap:0.15
2016-05-17 22:18:34,679 INFO resources.MemoryDistributor (MemoryDistributor.java:makeInitialAllocations(138)) - Informing: INPUT, 9C35570ADFCA474AAF1AE528C937CC58, org.apache.tez.mapreduce.input.MRInput: requested=0, allocated=0
2016-05-17 22:18:34,679 INFO resources.MemoryDistributor (MemoryDistributor.java:makeInitialAllocations(138)) - Informing: OUTPUT, F7D0FABC2C6A4B3B909C555D58A16B58, org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput: requested=104857600, allocated=104857600
2016-05-17 22:18:34,679 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(255)) - Starting Inputs/Outputs
2016-05-17 22:18:34,679 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(268)) - Input: 9C35570ADFCA474AAF1AE528C937CC58 being auto started by the framework. Subsequent instances will not be auto-started
2016-05-17 22:18:34,679 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(289)) - Num IOs determined for AutoStart: 1
2016-05-17 22:18:34,679 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(291)) - Waiting for 1 IOs to start
2016-05-17 22:18:34,680 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(419)) - Starting Input with src edge: 9C35570ADFCA474AAF1AE528C937CC58
2016-05-17 22:18:34,680 INFO input.MRInput (MRInput.java:start(450)) - MRInput setup to received events1
2016-05-17 22:18:34,680 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(421)) - Started Input with src edge: 9C35570ADFCA474AAF1AE528C937CC58
2016-05-17 22:18:34,680 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(304)) - AutoStartComplete
2016-05-17 22:18:34,681 INFO task.TezTaskRunner (TezTaskRunner.java:run(175)) - Running task, taskAttemptId=attempt_1463523514422_0001_1_00_000000_0
2016-05-17 22:18:34,682 INFO input.MRInput (MRInput.java:initFromEventInternal(619)) - Initializing RecordReader from event
2016-05-17 22:18:34,683 INFO input.MRInput (MRInput.java:initFromEventInternal(630)) - Split Details -> SplitClass: org.apache.hadoop.mapred.split.TezGroupedSplit, OldSplit: org.apache.hadoop.mapred.split.TezGroupedSplit@2e8058ee
2016-05-17 22:18:34,684 INFO graph.Hadoop2TezStreamGraph (Hadoop2TezStreamGraph.java:buildGraph(151)) - using streamed source: Hfs["TextDelimited[['num', 'lower']]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.5.x-tez-0.6.x/build/classes/test/data/lhs.txt"]
2016-05-17 22:18:34,684 INFO tez.FlowProcessor (FlowProcessor.java:run(121)) - sourcing from: Hfs["TextDelimited[['num', 'lower']]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.5.x-tez-0.6.x/build/classes/test/data/lhs.txt"] streamed: true, id: 9C35570ADFCA474AAF1AE528C937CC58
2016-05-17 22:18:34,684 INFO tez.FlowProcessor (FlowProcessor.java:run(124)) - sinking to: GroupBy(test)[by:[{1}:'num']], id: 6DC05B0DA886445CBD9C49D0844B10E9
2016-05-17 22:18:34,685 INFO element.TezGroupGate (TezGroupGate.java:prepare(98)) - calling OrderedPartitionedKVOutput#start() on: GroupBy(test)[by:[{1}:'num']] 6DC05B0DA886445CBD9C49D0844B10E9
2016-05-17 22:18:34,685 INFO lib.MRReaderMapred (MRReaderMapred.java:setIncrementalConfigParams(163)) - Processing split: org.apache.hadoop.mapred.split.TezGroupedSplit@2e8058ee
2016-05-17 22:18:34,685 INFO input.MRInput (MRInput.java:initFromEventInternal(634)) - Initialized RecordReader from event
2016-05-17 22:18:34,686 INFO input.MRInput (MRInput.java:processSplitEvent(586)) - Notifying on RecordReader Initialized
2016-05-17 22:18:34,689 INFO impl.ExternalSorter (ExternalSorter.java:<init>(167)) - keySerializer=cascading.tuple.hadoop.io.TuplePairSerializer@154b74da; valueSerializer=cascading.tuple.hadoop.io.ValueTupleSerializer@ce49f22; comparator=cascading.tuple.hadoop.util.ReverseGroupingSortingComparator@13c1b370; conf=cascading.tuple.hadoop.TupleSerialization,org.apache.hadoop.io.serializer.WritableSerialization,org.apache.hadoop.io.serializer.avro.AvroSpecificSerialization,org.apache.hadoop.io.serializer.avro.AvroReflectSerialization
2016-05-17 22:18:34,690 INFO impl.ExternalSorter (ExternalSorter.java:<init>(224)) - Instantiating Partitioner: [cascading.tuple.tez.util.GroupingSortingPartitioner]
2016-05-17 22:18:34,690 INFO common.TezRuntimeUtils (TezRuntimeUtils.java:instantiatePartitioner(109)) - Using partitioner class: cascading.tuple.tez.util.GroupingSortingPartitioner
2016-05-17 22:18:34,690 INFO common.TezRuntimeUtils (TezRuntimeUtils.java:instantiateCombiner(65)) - No combiner specified via tez.runtime.combiner.class. Combiner will not be used
2016-05-17 22:18:34,815 INFO dflt.DefaultSorter (DefaultSorter.java:setEquator(328)) - (EQUATOR) 0 kvi 26214396(104857584)
2016-05-17 22:18:34,816 INFO dflt.DefaultSorter (DefaultSorter.java:<init>(150)) - tez.runtime.io.sort.mb: 100
2016-05-17 22:18:34,816 INFO dflt.DefaultSorter (DefaultSorter.java:<init>(151)) - soft limit at 83886080
2016-05-17 22:18:34,816 INFO dflt.DefaultSorter (DefaultSorter.java:<init>(152)) - bufstart = 0; bufvoid = 104857600
2016-05-17 22:18:34,817 INFO dflt.DefaultSorter (DefaultSorter.java:<init>(153)) - kvstart = 26214396; length = 6553600
2016-05-17 22:18:34,817 INFO element.TezSourceStage (TezSourceStage.java:prepare(63)) - calling MRInput#start() on: Hfs["TextDelimited[['num', 'lower']]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.5.x-tez-0.6.x/build/classes/test/data/lhs.txt"]
2016-05-17 22:18:34,817 INFO input.MRInput (MRInput.java:start(450)) - MRInput setup to received events1
2016-05-17 22:18:34,818 INFO tez.FlowProcessor (FlowProcessor.java:waitForInputsReady(202)) - flow node id: C982EAFB40CC492D8D2F1992C1E8AEF9, all 1 inputs ready in: 00:00:00.000
2016-05-17 22:18:34,821 INFO task.TezTaskRunner (TezTaskRunner.java:run(177)) - Closing task, taskAttemptId=attempt_1463523514422_0001_1_00_000000_0
2016-05-17 22:18:34,822 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: C982EAFB40CC492D8D2F1992C1E8AEF9, mem on close (mb), free: 531, total: 654, max: 672
2016-05-17 22:18:34,822 INFO dflt.DefaultSorter (DefaultSorter.java:flush(589)) - Starting flush of map output
2016-05-17 22:18:34,822 INFO dflt.DefaultSorter (DefaultSorter.java:flush(607)) - Sorting & Spilling map output
2016-05-17 22:18:34,823 INFO dflt.DefaultSorter (DefaultSorter.java:flush(608)) - bufstart = 0; bufend = 286; bufvoid = 104857600
2016-05-17 22:18:34,823 INFO dflt.DefaultSorter (DefaultSorter.java:flush(610)) - kvstart = 26214396(104857584); kvend = 26214348(104857392); length = 49/6553600
2016-05-17 22:18:34,827 INFO dflt.DefaultSorter (DefaultSorter.java:spill(829)) - Finished spill 0
2016-05-17 22:18:34 Completed running task attempt: attempt_1463523514422_0001_1_00_000000_0
2016-05-17 22:18:34,831 INFO task.TezTaskRunner (TezTaskRunner.java:run(181)) - Task completed, taskAttemptId=attempt_1463523514422_0001_1_00_000000_0, fatalErrorOccurred=false
2016-05-17 22:18:34,832 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1463523514422_0001_1][Event:VERTEX_DATA_MOVEMENT_EVENTS_GENERATED]: vertexId=vertex_1463523514422_0001_1_00, eventCount=1
2016-05-17 22:18:34,833 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event TASK_ATTEMPT_FINISHED to history file
2016-05-17 22:18:34,833 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1463523514422_0001_1][Event:TASK_ATTEMPT_FINISHED]: vertexName=C982EAFB40CC492D8D2F1992C1E8AEF9, taskAttemptId=attempt_1463523514422_0001_1_00_000000_0, startTime=1463523514653, finishTime=1463523514832, timeTaken=179, status=SUCCEEDED, errorEnum=, diagnostics=, counters=Counters: 28, org.apache.tez.common.counters.DAGCounter, RACK_LOCAL_TASKS=1, File System Counters, FILE_BYTES_READ=110, FILE_BYTES_WRITTEN=354, FILE_READ_OPS=0, FILE_LARGE_READ_OPS=0, FILE_WRITE_OPS=0, org.apache.tez.common.counters.TaskCounter, SPILLED_RECORDS=13, GC_TIME_MILLIS=111, CPU_MILLISECONDS=0, PHYSICAL_MEMORY_BYTES=0, VIRTUAL_MEMORY_BYTES=0, COMMITTED_HEAP_BYTES=686292992, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=13, OUTPUT_BYTES=286, OUTPUT_BYTES_WITH_OVERHEAD=318, OUTPUT_BYTES_PHYSICAL=322, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=0, ADDITIONAL_SPILL_COUNT=0, cascading.flow.SliceCounters, Process_Begin_Time=1463523514818, Process_Duration=3, Process_End_Time=1463523514821, Read_Duration=3, Tuples_Read=13, Tuples_Written=13, Write_Duration=0, cascading.flow.StepCounters, Tuples_Read=13
2016-05-17 22:18:34,833 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:handle(692)) - attempt_1463523514422_0001_1_00_000000_0 TaskAttempt Transitioned from RUNNING to SUCCEEDED due to event TA_DONE
2016-05-17 22:18:34,833 INFO impl.TaskImpl (TaskImpl.java:transition(1052)) - Task succeeded with attempt attempt_1463523514422_0001_1_00_000000_0
2016-05-17 22:18:34,833 INFO rm.TaskSchedulerEventHandler (TaskSchedulerEventHandler.java:handleEvent(149)) - Processing the event EventType: S_TA_ENDED
2016-05-17 22:18:34,834 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:cleanup(712)) - Final Counters : Counters: 27 [[File System Counters FILE_BYTES_READ=110, FILE_BYTES_WRITTEN=354, FILE_READ_OPS=0, FILE_LARGE_READ_OPS=0, FILE_WRITE_OPS=0][org.apache.tez.common.counters.TaskCounter SPILLED_RECORDS=13, GC_TIME_MILLIS=111, CPU_MILLISECONDS=0, PHYSICAL_MEMORY_BYTES=0, VIRTUAL_MEMORY_BYTES=0, COMMITTED_HEAP_BYTES=686292992, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=13, OUTPUT_BYTES=286, OUTPUT_BYTES_WITH_OVERHEAD=318, OUTPUT_BYTES_PHYSICAL=322, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=0, ADDITIONAL_SPILL_COUNT=0][cascading.flow.SliceCounters Process_Begin_Time=1463523514818, Process_Duration=3, Process_End_Time=1463523514821, Read_Duration=3, Tuples_Read=13, Tuples_Written=13, Write_Duration=0][cascading.flow.StepCounters Tuples_Read=13]]
2016-05-17 22:18:34,834 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(164)) - Redirecting log file based on addend:
2016-05-17 22:18:34,834 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(179)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-17 22:18:34,835 INFO task.ContainerReporter (ContainerReporter.java:call(55)) - Attempting to fetch new task for container container_1463523514422_0001_00_000001
2016-05-17 22:18:34,834 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1463523514422_0001_1][Event:TASK_FINISHED]: vertexName=C982EAFB40CC492D8D2F1992C1E8AEF9, taskId=task_1463523514422_0001_1_00_000000, startTime=1463523514653, finishTime=1463523514833, timeTaken=180, status=SUCCEEDED, successfulAttemptID=attempt_1463523514422_0001_1_00_000000_0, diagnostics=, counters=Counters: 28, org.apache.tez.common.counters.DAGCounter, RACK_LOCAL_TASKS=1, File System Counters, FILE_BYTES_READ=110, FILE_BYTES_WRITTEN=354, FILE_READ_OPS=0, FILE_LARGE_READ_OPS=0, FILE_WRITE_OPS=0, org.apache.tez.common.counters.TaskCounter, SPILLED_RECORDS=13, GC_TIME_MILLIS=111, CPU_MILLISECONDS=0, PHYSICAL_MEMORY_BYTES=0, VIRTUAL_MEMORY_BYTES=0, COMMITTED_HEAP_BYTES=686292992, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=13, OUTPUT_BYTES=286, OUTPUT_BYTES_WITH_OVERHEAD=318, OUTPUT_BYTES_PHYSICAL=322, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=0, ADDITIONAL_SPILL_COUNT=0, cascading.flow.SliceCounters, Process_Begin_Time=1463523514818, Process_Duration=3, Process_End_Time=1463523514821, Read_Duration=3, Tuples_Read=13, Tuples_Written=13, Write_Duration=0, cascading.flow.StepCounters, Tuples_Read=13
2016-05-17 22:18:34,834 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event TASK_FINISHED to history file
2016-05-17 22:18:34,835 INFO impl.TaskImpl (TaskImpl.java:handle(862)) - task_1463523514422_0001_1_00_000000 Task Transitioned from RUNNING to SUCCEEDED due to event T_ATTEMPT_SUCCEEDED
2016-05-17 22:18:34,836 INFO impl.VertexImpl (VertexImpl.java:transition(3536)) - Num completed Tasks for vertex_1463523514422_0001_1_00 [C982EAFB40CC492D8D2F1992C1E8AEF9] : 1
2016-05-17 22:18:34,836 INFO impl.VertexImpl (VertexImpl.java:checkVertexForCompletion(1674)) - Vertex succeeded: vertex_1463523514422_0001_1_00 [C982EAFB40CC492D8D2F1992C1E8AEF9]
2016-05-17 22:18:34,838 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event VERTEX_FINISHED to history file
2016-05-17 22:18:34,838 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1463523514422_0001_1][Event:VERTEX_FINISHED]: vertexName=C982EAFB40CC492D8D2F1992C1E8AEF9, vertexId=vertex_1463523514422_0001_1_00, initRequestedTime=1463523514598, initedTime=1463523514632, startRequestedTime=1463523514633, startedTime=1463523514633, finishTime=1463523514836, timeTaken=203, status=SUCCEEDED, diagnostics=, counters=Counters: 28, org.apache.tez.common.counters.DAGCounter, RACK_LOCAL_TASKS=1, File System Counters, FILE_BYTES_READ=110, FILE_BYTES_WRITTEN=354, FILE_READ_OPS=0, FILE_LARGE_READ_OPS=0, FILE_WRITE_OPS=0, org.apache.tez.common.counters.TaskCounter, SPILLED_RECORDS=13, GC_TIME_MILLIS=111, CPU_MILLISECONDS=0, PHYSICAL_MEMORY_BYTES=0, VIRTUAL_MEMORY_BYTES=0, COMMITTED_HEAP_BYTES=686292992, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=13, OUTPUT_BYTES=286, OUTPUT_BYTES_WITH_OVERHEAD=318, OUTPUT_BYTES_PHYSICAL=322, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=0, ADDITIONAL_SPILL_COUNT=0, cascading.flow.SliceCounters, Process_Begin_Time=1463523514818, Process_Duration=3, Process_End_Time=1463523514821, Read_Duration=3, Tuples_Read=13, Tuples_Written=13, Write_Duration=0, cascading.flow.StepCounters, Tuples_Read=13, vertexStats=firstTaskStartTime=1463523514653, firstTasksToStart=[ task_1463523514422_0001_1_00_000000 ], lastTaskFinishTime=1463523514832, lastTasksToFinish=[ task_1463523514422_0001_1_00_000000 ], minTaskDuration=179, maxTaskDuration=179, avgTaskDuration=179.0, numSuccessfulTasks=1, shortestDurationTasks=[ task_1463523514422_0001_1_00_000000 ], longestDurationTasks=[ task_1463523514422_0001_1_00_000000 ], vertexTaskStats={numFailedTasks=0, numSucceededTasks=1, numKilledTaskAttempts=0, numKilledTasks=0, numFailedTaskAttempts=0, numCompletedTasks=1}
2016-05-17 22:18:34,838 INFO impl.VertexImpl (VertexImpl.java:handle(1564)) - vertex_1463523514422_0001_1_00 [C982EAFB40CC492D8D2F1992C1E8AEF9] transitioned from RUNNING to SUCCEEDED due to event V_TASK_COMPLETED
2016-05-17 22:18:34,839 INFO container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1463523514422_0001_00_000001 transitioned from RUNNING to IDLE via event C_TA_SUCCEEDED
2016-05-17 22:18:34,839 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1463523514422_0001_1][Event:CONTAINER_STOPPED]: containerId=container_1463523514422_0001_00_000001, stoppedTime=1463523514839, exitStatus=0
2016-05-17 22:18:34,839 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event CONTAINER_STOPPED to history file
2016-05-17 22:18:34,839 INFO container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1463523514422_0001_00_000001 transitioned from IDLE to STOP_REQUESTED via event C_STOP_REQUEST
2016-05-17 22:18:34,840 INFO impl.VertexImpl (VertexImpl.java:transition(3477)) - Source task attempt completed for vertex: vertex_1463523514422_0001_1_01 [F7D0FABC2C6A4B3B909C555D58A16B58] attempt: attempt_1463523514422_0001_1_00_000000_0 with state: SUCCEEDED vertexState: RUNNING
2016-05-17 22:18:34,840 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:schedulePendingTasks(608)) - All source tasks assigned. Ramping up 1 remaining tasks for vertex: F7D0FABC2C6A4B3B909C555D58A16B58
2016-05-17 22:18:34,840 INFO impl.VertexImpl (VertexImpl.java:scheduleTasks(1250)) - Routing pending task events for vertex: vertex_1463523514422_0001_1_01 [F7D0FABC2C6A4B3B909C555D58A16B58]
2016-05-17 22:18:34,841 INFO impl.DAGImpl (DAGImpl.java:transition(1747)) - Vertex vertex_1463523514422_0001_1_00 [C982EAFB40CC492D8D2F1992C1E8AEF9] completed., numCompletedVertices=1, numSuccessfulVertices=1, numFailedVertices=0, numKilledVertices=0, numVertices=2
2016-05-17 22:18:34,841 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:stop(244)) - Ignoring stop request for containerId container_1463523514422_0001_00_000001. Relying on regular task shutdown for it to end
2016-05-17 22:18:34,841 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1463523514422_0001_1][Event:TASK_STARTED]: vertexName=F7D0FABC2C6A4B3B909C555D58A16B58, taskId=task_1463523514422_0001_1_01_000000, scheduledTime=1463523514841, launchTime=1463523514841
2016-05-17 22:18:34,841 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event TASK_STARTED to history file
2016-05-17 22:18:34,842 INFO impl.TaskImpl (TaskImpl.java:handle(862)) - task_1463523514422_0001_1_01_000000 Task Transitioned from NEW to SCHEDULED due to event T_SCHEDULE
2016-05-17 22:18:34,842 INFO impl.DAGSchedulerNaturalOrderControlled (DAGSchedulerNaturalOrderControlled.java:scheduleTask(111)) - Scheduled vertex: vertex_1463523514422_0001_1_01 [F7D0FABC2C6A4B3B909C555D58A16B58]
2016-05-17 22:18:34,842 INFO container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1463523514422_0001_00_000001 transitioned from STOP_REQUESTED to STOPPING via event C_NM_STOP_SENT
2016-05-17 22:18:34,843 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:transition(1014)) - remoteTaskSpec:DAGName : [C886F761D5D24A6BA33C0106FE67C18A/DB6F7C03C9E6433D8612EA31AE545BC1] test/(1/1) ...irstNBufferReverse/firstn, VertexName: F7D0FABC2C6A4B3B909C555D58A16B58, VertexParallelism: 1, TaskAttemptID:attempt_1463523514422_0001_1_01_000000_0, processorName=cascading.flow.tez.FlowProcessor, inputSpecListSize=1, outputSpecListSize=1, inputSpecList=[{{ sourceVertexName=C982EAFB40CC492D8D2F1992C1E8AEF9, physicalEdgeCount=1, inputClassName=org.apache.tez.runtime.library.input.OrderedGroupedKVInput }}, ], outputSpecList=[{{ destinationVertexName=8C05B316B49B4068837D5FC66B1D2E75, physicalEdgeCount=0, outputClassName=org.apache.tez.mapreduce.output.MROutput }}, ]
2016-05-17 22:18:34,843 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:handle(692)) - attempt_1463523514422_0001_1_01_000000_0 TaskAttempt Transitioned from NEW to START_WAIT due to event TA_SCHEDULE
2016-05-17 22:18:34,843 INFO rm.TaskSchedulerEventHandler (TaskSchedulerEventHandler.java:handleEvent(149)) - Processing the event EventType: S_TA_LAUNCH_REQUEST
2016-05-17 22:18:34,844 INFO container.AMContainerImpl (AMContainerImpl.java:transition(521)) - Sending Launch Request for Container with id: container_1463523514422_0001_00_000002
2016-05-17 22:18:34,844 INFO container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1463523514422_0001_00_000002 transitioned from ALLOCATED to LAUNCHING via event C_LAUNCH_REQUEST
2016-05-17 22:18:34,845 INFO app.TaskAttemptListenerImpTezDag (TaskAttemptListenerImpTezDag.java:getTask(213)) - Container with id: container_1463523514422_0001_00_000001 is valid, but no longer registered, and will be killed
2016-05-17 22:18:34,845 INFO task.ContainerReporter (ContainerReporter.java:call(65)) - Got TaskUpdate for containerId= container_1463523514422_0001_00_000001: 10 ms after starting to poll. TaskInfo: shouldDie: true
2016-05-17 22:18:34,845 INFO task.TezChild (TezChild.java:run(207)) - ContainerTask returned shouldDie=true for container container_1463523514422_0001_00_000001, Exiting
2016-05-17 22:18:34,845 INFO task.TezChild (TezChild.java:shutdown(343)) - Shutdown invoked for container container_1463523514422_0001_00_000001
2016-05-17 22:18:34,846 INFO task.TezChild (TezChild.java:newTezChild(432)) - PID, containerIdentifier: null, container_1463523514422_0001_00_000002
2016-05-17 22:18:34,846 WARN impl.MetricsSystemImpl (MetricsSystemImpl.java:init(145)) - DAGAppMaster metrics system already initialized!
2016-05-17 22:18:34,846 INFO task.TezChild (TezChild.java:shutdown(345)) - Shutting down container container_1463523514422_0001_00_000001
2016-05-17 22:18:34,848 INFO container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1463523514422_0001_00_000002 transitioned from LAUNCHING to IDLE via event C_LAUNCHED
2016-05-17 22:18:34,849 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1463523514422_0001_1][Event:CONTAINER_LAUNCHED]: containerId=container_1463523514422_0001_00_000002, launchTime=1463523514848
2016-05-17 22:18:34,849 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event CONTAINER_LAUNCHED to history file
2016-05-17 22:18:34,849 INFO task.ContainerReporter (ContainerReporter.java:call(55)) - Attempting to fetch new task for container container_1463523514422_0001_00_000002
2016-05-17 22:18:34,850 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(270)) - ContainerExecutionResult for: container_1463523514422_0001_00_000001 = ContainerExecutionResult{exitStatus=SUCCESS, throwable=null, errorMessage='Asked to die by the AM'}
2016-05-17 22:18:34,850 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(274)) - Container: container_1463523514422_0001_00_000001 completed successfully
2016-05-17 22:18:34,850 INFO container.AMContainerImpl (AMContainerImpl.java:transition(827)) - Assigned taskAttempt + [attempt_1463523514422_0001_1_01_000000_0] to container: [container_1463523514422_0001_00_000002]
2016-05-17 22:18:34,851 INFO container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1463523514422_0001_00_000002 transitioned from IDLE to RUNNING via event C_PULL_TA
2016-05-17 22:18:34,851 INFO container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1463523514422_0001_00_000001 transitioned from STOPPING to COMPLETED via event C_COMPLETED
2016-05-17 22:18:34,852 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:transition(1146)) - TaskAttempt: [attempt_1463523514422_0001_1_01_000000_0] started. Is using containerId: [container_1463523514422_0001_00_000002] on NM: [127.0.0.1:0]
2016-05-17 22:18:34,852 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1463523514422_0001_1][Event:TASK_ATTEMPT_STARTED]: vertexName=F7D0FABC2C6A4B3B909C555D58A16B58, taskAttemptId=attempt_1463523514422_0001_1_01_000000_0, startTime=1463523514852, containerId=container_1463523514422_0001_00_000002, nodeId=127.0.0.1:0, inProgressLogs=127.0.0.1:0/node/containerlogs/container_1463523514422_0001_00_000002/teamcity, completedLogs=
2016-05-17 22:18:34,852 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:handle(692)) - attempt_1463523514422_0001_1_01_000000_0 TaskAttempt Transitioned from START_WAIT to RUNNING due to event TA_STARTED_REMOTELY
2016-05-17 22:18:34,853 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event TASK_ATTEMPT_STARTED to history file
2016-05-17 22:18:34,853 INFO impl.TaskImpl (TaskImpl.java:handle(862)) - task_1463523514422_0001_1_01_000000 Task Transitioned from SCHEDULED to RUNNING due to event T_ATTEMPT_LAUNCHED
2016-05-17 22:18:34,853 INFO app.TaskAttemptListenerImpTezDag (TaskAttemptListenerImpTezDag.java:getTask(240)) - Container with id: container_1463523514422_0001_00_000002 given task: attempt_1463523514422_0001_1_01_000000_0
2016-05-17 22:18:34,853 INFO task.ContainerReporter (ContainerReporter.java:call(65)) - Got TaskUpdate for containerId= container_1463523514422_0001_00_000002: 0 ms after starting to poll. TaskInfo: shouldDie: false, currentTaskAttemptId: attempt_1463523514422_0001_1_01_000000_0
2016-05-17 22:18:34 Starting to run new task attempt: attempt_1463523514422_0001_1_01_000000_0
2016-05-17 22:18:34,854 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(164)) - Redirecting log file based on addend: attempt_1463523514422_0001_1_01_000000_0
2016-05-17 22:18:34,854 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(179)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-17 22:18:34,854 INFO task.TezChild (TezChild.java:handleNewTaskCredentials(273)) - Refreshing UGI since Credentials have changed
2016-05-17 22:18:34,854 INFO task.TezChild (TezChild.java:handleNewTaskCredentials(276)) - Credentials : #Tokens=1, #SecretKeys=0
2016-05-17 22:18:34,854 INFO task.TezChild (TezChild.java:handleNewTaskLocalResources(302)) - Localizing additional local resources for Task : {}
2016-05-17 22:18:34,854 INFO task.TezChild (TezChild.java:handleNewTaskLocalResources(312)) - Done localizing additional resources
2016-05-17 22:18:34,855 INFO metrics.TaskCounterUpdater (TaskCounterUpdater.java:initResourceCalculatorPlugin(148)) - Using ResourceCalculatorProcessTree : [ ]
2016-05-17 22:18:34,855 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:<init>(149)) - Initializing LogicalIOProcessorRuntimeTask with TaskSpec: DAGName : [C886F761D5D24A6BA33C0106FE67C18A/DB6F7C03C9E6433D8612EA31AE545BC1] test/(1/1) ...irstNBufferReverse/firstn, VertexName: F7D0FABC2C6A4B3B909C555D58A16B58, VertexParallelism: 1, TaskAttemptID:attempt_1463523514422_0001_1_01_000000_0, processorName=cascading.flow.tez.FlowProcessor, inputSpecListSize=1, outputSpecListSize=1, inputSpecList=[{{ sourceVertexName=C982EAFB40CC492D8D2F1992C1E8AEF9, physicalEdgeCount=1, inputClassName=org.apache.tez.runtime.library.input.OrderedGroupedKVInput }}, ], outputSpecList=[{{ destinationVertexName=8C05B316B49B4068837D5FC66B1D2E75, physicalEdgeCount=0, outputClassName=org.apache.tez.mapreduce.output.MROutput }}, ]
2016-05-17 22:18:34,855 INFO resources.MemoryDistributor (MemoryDistributor.java:<init>(85)) - InitialMemoryDistributor (isEnabled=true) invoked with: numInputs=1, numOutputs=1, JVM.maxFree=704643072
2016-05-17 22:18:34,856 INFO task.TezTaskRunner (TezTaskRunner.java:run(172)) - Initializing task, taskAttemptId=attempt_1463523514422_0001_1_01_000000_0
2016-05-17 22:18:34,857 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(191)) - Initializing LogicalProcessorIORuntimeTask
2016-05-17 22:18:34,857 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(195)) - Creating processor, processorClassName=cascading.flow.tez.FlowProcessor
2016-05-17 22:18:34,856 INFO impl.TaskImpl (TaskImpl.java:getTaskAttemptTezEvents(481)) - TaskAttempt:attempt_1463523514422_0001_1_01_000000_0 sent events: (0-1)
2016-05-17 22:18:34,858 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(389)) - Initializing Input using InputSpec: { sourceVertexName=C982EAFB40CC492D8D2F1992C1E8AEF9, physicalEdgeCount=1, inputClassName=org.apache.tez.runtime.library.input.OrderedGroupedKVInput }
2016-05-17 22:18:34,858 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(438)) - Initializing Output using OutputSpec: { destinationVertexName=8C05B316B49B4068837D5FC66B1D2E75, physicalEdgeCount=0, outputClassName=org.apache.tez.mapreduce.output.MROutput }
2016-05-17 22:18:34,858 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initializeLogicalIOProcessor(488)) - Initializing processor, processorClassName=cascading.flow.tez.FlowProcessor
2016-05-17 22:18:34,858 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:createInput(537)) - Creating Input
2016-05-17 22:18:34,859 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:createOutput(565)) - Creating Output
2016-05-17 22:18:34,859 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(397)) - Initializing Input with src edge: C982EAFB40CC492D8D2F1992C1E8AEF9
2016-05-17 22:18:34,859 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(446)) - Initializing Output with dest edge: 8C05B316B49B4068837D5FC66B1D2E75
2016-05-17 22:18:34,859 INFO output.MROutput (MROutput.java:initialize(347)) - Initializing Simple Output
2016-05-17 22:18:34,860 INFO resources.MemoryDistributor$RequestorInfo (MemoryDistributor.java:<init>(233)) - Received request: 0, type: OUTPUT, componentVertexName: 8C05B316B49B4068837D5FC66B1D2E75
2016-05-17 22:18:34,861 INFO orderedgrouped.MergeManager (MergeManager.java:getInitialMemoryRequirement(310)) - Initial Shuffle Memory Required: 634178752, based on INPUT_BUFFER_factor: 0.9
2016-05-17 22:18:34,861 INFO orderedgrouped.MergeManager (MergeManager.java:getInitialMemoryRequirement(320)) - Initial Memory required for final merged output: 0, using factor: 0.0
2016-05-17 22:18:34,861 INFO resources.MemoryDistributor$RequestorInfo (MemoryDistributor.java:<init>(233)) - Received request: 634178752, type: INPUT, componentVertexName: C982EAFB40CC492D8D2F1992C1E8AEF9
2016-05-17 22:18:34,861 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(403)) - Initialized Input with src edge: C982EAFB40CC492D8D2F1992C1E8AEF9
2016-05-17 22:18:34,863 INFO tez.FlowProcessor (FlowProcessor.java:initialize(88)) - cascading version: 3.1.0
2016-05-17 22:18:34,871 INFO tez.FlowProcessor (FlowProcessor.java:initialize(94)) - flow node id: F7D0FABC2C6A4B3B909C555D58A16B58, ordinal: 1
2016-05-17 22:18:34,872 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: F7D0FABC2C6A4B3B909C555D58A16B58, mem on start (mb), free: 525, total: 654, max: 672
2016-05-17 22:18:34,872 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initializeLogicalIOProcessor(491)) - Initialized processor, processorClassName=cascading.flow.tez.FlowProcessor
2016-05-17 22:18:34,872 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(220)) - Waiting for 2 initializers to finish
2016-05-17 22:18:34,872 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(220)) - Waiting for 1 initializers to finish
2016-05-17 22:18:34,872 INFO output.MROutput (MROutput.java:initialize(418)) - Initialized Simple Output, using_new_api: false
2016-05-17 22:18:34,873 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(452)) - Initialized Output with dest edge: 8C05B316B49B4068837D5FC66B1D2E75
2016-05-17 22:18:34,873 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(233)) - All initializers finished
2016-05-17 22:18:34,873 INFO resources.MemoryDistributor (MemoryDistributor.java:makeInitialAllocations(124)) - Using Allocator class: org.apache.tez.runtime.library.resources.WeightedScalingMemoryDistributor
2016-05-17 22:18:34,873 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:getRequestTypeForClass(197)) - Falling back to RequestType.OTHER for class: org.apache.tez.mapreduce.output.MROutput
2016-05-17 22:18:34,874 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:initialProcessMemoryRequestContext(171)) - ScaleFactor: 1, for type: OTHER
2016-05-17 22:18:34,874 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:initialProcessMemoryRequestContext(171)) - ScaleFactor: 12, for type: SORTED_MERGED_INPUT
2016-05-17 22:18:34,874 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:computeReservedFraction(262)) - InitialReservationFraction=0.3, AdditionalReservationFractionForIOs=0.03, finalReserveFractionUsed=0.32999999999999996
2016-05-17 22:18:34,874 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:assignMemory(126)) - Scaling Requests. NumRequests: 2, numScaledRequests: 13, TotalRequested: 634178752, TotalRequestedScaled: 5.853957710769231E8, TotalJVMHeap: 704643072, TotalAvailable: 472110858, TotalRequested/TotalJVMHeap:0.90
2016-05-17 22:18:34,875 INFO resources.MemoryDistributor (MemoryDistributor.java:makeInitialAllocations(138)) - Informing: OUTPUT, 8C05B316B49B4068837D5FC66B1D2E75, org.apache.tez.mapreduce.output.MROutput: requested=0, allocated=0
2016-05-17 22:18:34,875 INFO resources.MemoryDistributor (MemoryDistributor.java:makeInitialAllocations(138)) - Informing: INPUT, C982EAFB40CC492D8D2F1992C1E8AEF9, org.apache.tez.runtime.library.input.OrderedGroupedKVInput: requested=634178752, allocated=472110858
2016-05-17 22:18:34,875 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(255)) - Starting Inputs/Outputs
2016-05-17 22:18:34,875 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(268)) - Input: C982EAFB40CC492D8D2F1992C1E8AEF9 being auto started by the framework. Subsequent instances will not be auto-started
2016-05-17 22:18:34,875 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(419)) - Starting Input with src edge: C982EAFB40CC492D8D2F1992C1E8AEF9
2016-05-17 22:18:34,876 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(289)) - Num IOs determined for AutoStart: 1
2016-05-17 22:18:34,876 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(291)) - Waiting for 1 IOs to start
2016-05-17 22:18:34,876 INFO common.TezRuntimeUtils (TezRuntimeUtils.java:instantiateCombiner(65)) - No combiner specified via tez.runtime.combiner.class. Combiner will not be used
2016-05-17 22:18:34,877 INFO orderedgrouped.Shuffle (Shuffle.java:<init>(182)) - Shuffle assigned with 1 inputs, codec: NoneifileReadAhead: true
2016-05-17 22:18:34,877 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:<init>(158)) - ShuffleScheduler running for sourceVertex: C982EAFB40CC492D8D2F1992C1E8AEF9 with configuration: maxFetchFailuresBeforeReporting=5, reportReadErrorImmediately=true, maxFailedUniqueFetches=1, abortFailureLimit=30, maxMapRuntime=0, maxTaskOutputAtOnce=20
2016-05-17 22:18:34,877 INFO orderedgrouped.MergeManager (MergeManager.java:<init>(221)) - InitialRequest: ShuffleMem=634178752, postMergeMem=0, RuntimeTotalAvailable=472110858. Updated to: ShuffleMem=472110858, postMergeMem: 0
2016-05-17 22:18:34,877 INFO orderedgrouped.MergeManager (MergeManager.java:<init>(252)) - MergerManager: memoryLimit=472110858, maxSingleShuffleLimit=118027712, mergeThreshold=424899744, ioSortFactor=100, memToMemMergeOutputsThreshold=100
2016-05-17 22:18:34,877 INFO orderedgrouped.Shuffle (Shuffle.java:<init>(233)) - Num fetchers being started: 1
2016-05-17 22:18:34,879 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(421)) - Started Input with src edge: C982EAFB40CC492D8D2F1992C1E8AEF9
2016-05-17 22:18:34,879 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(304)) - AutoStartComplete
2016-05-17 22:18:34,879 INFO task.TezTaskRunner (TezTaskRunner.java:run(175)) - Running task, taskAttemptId=attempt_1463523514422_0001_1_01_000000_0
2016-05-17 22:18:34,880 INFO orderedgrouped.ShuffleInputEventHandlerOrderedGrouped (ShuffleInputEventHandlerOrderedGrouped.java:processDataMovementEvent(84)) - DME srcIdx: 0, targetIdx: 0, attemptNum: 0, payload: [host: ip-10-123-148-208, port: 0, pathComponent: attempt_1463523514422_0001_1_00_000000_0_10043, runDuration: 163831, ]
2016-05-17 22:18:34,882 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:logIndividualFetchComplete(250)) - Completed fetch for attempt: InputAttemptIdentifier [inputIdentifier=InputIdentifier [inputIndex=0], attemptNumber=0, pathComponent=attempt_1463523514422_0001_1_00_000000_0_10043] to DISK_DIRECT, CompressedSize=322, DecompressedSize=318,EndTime=1463523514882, TimeTaken=1, Rate=0.31 MB/s
2016-05-17 22:18:34,882 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:copySucceeded(213)) - All inputs fetched for input vertex : C982EAFB40CC492D8D2F1992C1E8AEF9
2016-05-17 22:18:34,882 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:logProgress(263)) - copy(1 of 1. Transfer rate (CumulativeDataFetched/TimeSinceInputStarted)) 0.00 MB/s)
2016-05-17 22:18:34,883 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:freeHost(529)) - ip-10-123-148-208:0 freed by fetcher [C982EAFB40CC492D8D2F1992C1E8AEF9] #8 in 2ms
2016-05-17 22:18:34,883 INFO orderedgrouped.Shuffle (Shuffle.java:cleanupFetchers(394)) - Shutdown..fetcher [C982EAFB40CC492D8D2F1992C1E8AEF9] #8, status:false, isInterrupted:false
2016-05-17 22:18:34,883 INFO graph.Hadoop2TezStreamGraph (Hadoop2TezStreamGraph.java:buildGraph(151)) - using streamed source: GroupBy(test)[by:[{1}:'num']]
2016-05-17 22:18:34,884 INFO tez.FlowProcessor (FlowProcessor.java:run(121)) - sourcing from: GroupBy(test)[by:[{1}:'num']] streamed: true, id: 6DC05B0DA886445CBD9C49D0844B10E9
2016-05-17 22:18:34,884 INFO tez.FlowProcessor (FlowProcessor.java:run(124)) - sinking to: Hfs["TextDelimited[['num', 'lower']]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.5.x-tez-0.6.x/build/test/output/hadoop2-tez/bufferpipesplatform/testFirstNBufferReverse/firstn"], id: 8C05B316B49B4068837D5FC66B1D2E75
2016-05-17 22:18:34,884 INFO element.TezSinkStage (TezSinkStage.java:prepare(57)) - calling MROutput#start() on: Hfs["TextDelimited[['num', 'lower']]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.5.x-tez-0.6.x/build/test/output/hadoop2-tez/bufferpipesplatform/testFirstNBufferReverse/firstn"]
2016-05-17 22:18:34,884 INFO element.TezGroupGate (TezGroupGate.java:prepare(90)) - calling OrderedGroupedKVInput#start() on: GroupBy(test)[by:[{1}:'num']] 6DC05B0DA886445CBD9C49D0844B10E9, for 1 inputs
2016-05-17 22:18:34,885 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(854)) - finalMerge called with 0 in-memory map-outputs and 1 on-disk map-outputs
2016-05-17 22:18:34,885 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(951)) - Merging 1 files, 322 bytes from disk
2016-05-17 22:18:34,886 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(966)) - Merging 0 segments, 0 bytes from memory into reduce
2016-05-17 22:18:34,886 INFO impl.TezMerger (TezMerger.java:merge(560)) - Merging 1 sorted segments
2016-05-17 22:18:34,886 INFO impl.TezMerger (TezMerger.java:merge(665)) - Down to the last merge-pass, with 1 segments left of total size: 298 bytes
2016-05-17 22:18:34,886 INFO orderedgrouped.Shuffle (Shuffle.java:call(381)) - merge complete for input vertex : C982EAFB40CC492D8D2F1992C1E8AEF9
2016-05-17 22:18:34,886 INFO orderedgrouped.Shuffle (Shuffle.java:onSuccess(488)) - Shuffle Runner thread complete
2016-05-17 22:18:34,886 INFO tez.FlowProcessor (FlowProcessor.java:waitForInputsReady(202)) - flow node id: F7D0FABC2C6A4B3B909C555D58A16B58, all 1 inputs ready in: 00:00:00.001
2016-05-17 22:18:34,888 INFO common.ValuesIterator (ValuesIterator.java:<init>(83)) - keyDeserializer=cascading.tuple.hadoop.io.TuplePairDeserializer@384eef0; valueDeserializer=cascading.tuple.hadoop.io.ValueTupleDeserializer@c62d05e
2016-05-17 22:18:34,890 INFO output.MROutput (MROutput.java:flush(534)) - Flushing Simple Output
2016-05-17 22:18:34,890 INFO output.MROutput (MROutput.java:flush(544)) - Flushed Simple Output
2016-05-17 22:18:34,891 INFO app.TaskAttemptListenerImpTezDag (TaskAttemptListenerImpTezDag.java:canCommit(353)) - Commit go/no-go request from attempt_1463523514422_0001_1_01_000000_0
2016-05-17 22:18:34,891 INFO impl.TaskImpl (TaskImpl.java:canCommit(750)) - attempt_1463523514422_0001_1_01_000000_0 given a go for committing the task output.
2016-05-17 22:18:34,892 INFO task.TezTaskRunner (TezTaskRunner.java:run(177)) - Closing task, taskAttemptId=attempt_1463523514422_0001_1_01_000000_0
2016-05-17 22:18:34,892 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: F7D0FABC2C6A4B3B909C555D58A16B58, mem on close (mb), free: 522, total: 654, max: 672
2016-05-17 22:18:34,892 INFO orderedgrouped.Shuffle (Shuffle.java:shutdown(320)) - Shutting down Shuffle for source: C982EAFB40CC492D8D2F1992C1E8AEF9
2016-05-17 22:18:34,892 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(854)) - finalMerge called with 0 in-memory map-outputs and 0 on-disk map-outputs
2016-05-17 22:18:34,893 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(951)) - Merging 0 files, 0 bytes from disk
2016-05-17 22:18:34,894 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(966)) - Merging 0 segments, 0 bytes from memory into reduce
2016-05-17 22:18:34,894 INFO impl.TezMerger (TezMerger.java:merge(560)) - Merging 0 sorted segments
2016-05-17 22:18:34,894 INFO impl.TezMerger (TezMerger.java:merge(562)) - Nothing to merge. Returning an empty iterator
2016-05-17 22:18:34 Completed running task attempt: attempt_1463523514422_0001_1_01_000000_0
2016-05-17 22:18:34,895 INFO task.TezTaskRunner (TezTaskRunner.java:run(181)) - Task completed, taskAttemptId=attempt_1463523514422_0001_1_01_000000_0, fatalErrorOccurred=false
2016-05-17 22:18:34,896 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1463523514422_0001_1][Event:TASK_ATTEMPT_FINISHED]: vertexName=F7D0FABC2C6A4B3B909C555D58A16B58, taskAttemptId=attempt_1463523514422_0001_1_01_000000_0, startTime=1463523514852, finishTime=1463523514896, timeTaken=44, status=SUCCEEDED, errorEnum=, diagnostics=, counters=Counters: 46, File System Counters, FILE_BYTES_READ=354, FILE_BYTES_WRITTEN=32, FILE_READ_OPS=0, FILE_LARGE_READ_OPS=0, FILE_WRITE_OPS=0, org.apache.tez.common.counters.TaskCounter, REDUCE_INPUT_GROUPS=12, REDUCE_INPUT_RECORDS=5, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=13, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, GC_TIME_MILLIS=0, CPU_MILLISECONDS=0, PHYSICAL_MEMORY_BYTES=0, VIRTUAL_MEMORY_BYTES=0, COMMITTED_HEAP_BYTES=686292992, OUTPUT_RECORDS=5, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=322, SHUFFLE_BYTES=322, SHUFFLE_BYTES_DECOMPRESSED=318, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=322, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=5, MERGE_PHASE_TIME=9, FIRST_EVENT_RECEIVED=4, LAST_EVENT_RECEIVED=4, Shuffle Errors, BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0, cascading.flow.SliceCounters, Process_Begin_Time=1463523514887, Process_Duration=5, Process_End_Time=1463523514892, Read_Duration=1, Tuples_Read=5, Tuples_Written=5, Write_Duration=0, cascading.flow.StepCounters, Tuples_Written=5
2016-05-17 22:18:34,897 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:handle(692)) - attempt_1463523514422_0001_1_01_000000_0 TaskAttempt Transitioned from RUNNING to SUCCEEDED due to event TA_DONE
2016-05-17 22:18:34,897 INFO impl.TaskImpl (TaskImpl.java:transition(1052)) - Task succeeded with attempt attempt_1463523514422_0001_1_01_000000_0
2016-05-17 22:18:34,897 INFO rm.TaskSchedulerEventHandler (TaskSchedulerEventHandler.java:handleEvent(149)) - Processing the event EventType: S_TA_ENDED
2016-05-17 22:18:34,897 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:cleanup(712)) - Final Counters : Counters: 46 [[File System Counters FILE_BYTES_READ=354, FILE_BYTES_WRITTEN=32, FILE_READ_OPS=0, FILE_LARGE_READ_OPS=0, FILE_WRITE_OPS=0][org.apache.tez.common.counters.TaskCounter REDUCE_INPUT_GROUPS=12, REDUCE_INPUT_RECORDS=5, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=13, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, GC_TIME_MILLIS=0, CPU_MILLISECONDS=0, PHYSICAL_MEMORY_BYTES=0, VIRTUAL_MEMORY_BYTES=0, COMMITTED_HEAP_BYTES=686292992, OUTPUT_RECORDS=5, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=322, SHUFFLE_BYTES=322, SHUFFLE_BYTES_DECOMPRESSED=318, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=322, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=5, MERGE_PHASE_TIME=9, FIRST_EVENT_RECEIVED=4, LAST_EVENT_RECEIVED=4][Shuffle Errors BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0][cascading.flow.SliceCounters Process_Begin_Time=1463523514887, Process_Duration=5, Process_End_Time=1463523514892, Read_Duration=1, Tuples_Read=5, Tuples_Written=5, Write_Duration=0][cascading.flow.StepCounters Tuples_Written=5]]
2016-05-17 22:18:34,897 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1463523514422_0001_1][Event:TASK_FINISHED]: vertexName=F7D0FABC2C6A4B3B909C555D58A16B58, taskId=task_1463523514422_0001_1_01_000000, startTime=1463523514852, finishTime=1463523514897, timeTaken=45, status=SUCCEEDED, successfulAttemptID=attempt_1463523514422_0001_1_01_000000_0, diagnostics=, counters=Counters: 46, File System Counters, FILE_BYTES_READ=354, FILE_BYTES_WRITTEN=32, FILE_READ_OPS=0, FILE_LARGE_READ_OPS=0, FILE_WRITE_OPS=0, org.apache.tez.common.counters.TaskCounter, REDUCE_INPUT_GROUPS=12, REDUCE_INPUT_RECORDS=5, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=13, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, GC_TIME_MILLIS=0, CPU_MILLISECONDS=0, PHYSICAL_MEMORY_BYTES=0, VIRTUAL_MEMORY_BYTES=0, COMMITTED_HEAP_BYTES=686292992, OUTPUT_RECORDS=5, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=322, SHUFFLE_BYTES=322, SHUFFLE_BYTES_DECOMPRESSED=318, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=322, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=5, MERGE_PHASE_TIME=9, FIRST_EVENT_RECEIVED=4, LAST_EVENT_RECEIVED=4, Shuffle Errors, BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0, cascading.flow.SliceCounters, Process_Begin_Time=1463523514887, Process_Duration=5, Process_End_Time=1463523514892, Read_Duration=1, Tuples_Read=5, Tuples_Written=5, Write_Duration=0, cascading.flow.StepCounters, Tuples_Written=5
2016-05-17 22:18:34,897 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(164)) - Redirecting log file based on addend:
2016-05-17 22:18:34,897 INFO impl.TaskImpl (TaskImpl.java:handle(862)) - task_1463523514422_0001_1_01_000000 Task Transitioned from RUNNING to SUCCEEDED due to event T_ATTEMPT_SUCCEEDED
2016-05-17 22:18:34,898 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(179)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-17 22:18:34,898 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event TASK_ATTEMPT_FINISHED to history file
2016-05-17 22:18:34,898 INFO task.ContainerReporter (ContainerReporter.java:call(55)) - Attempting to fetch new task for container container_1463523514422_0001_00_000002
2016-05-17 22:18:34,898 INFO impl.VertexImpl (VertexImpl.java:transition(3536)) - Num completed Tasks for vertex_1463523514422_0001_1_01 [F7D0FABC2C6A4B3B909C555D58A16B58] : 1
2016-05-17 22:18:34,898 INFO impl.VertexImpl (VertexImpl.java:checkVertexForCompletion(1674)) - Vertex succeeded: vertex_1463523514422_0001_1_01 [F7D0FABC2C6A4B3B909C555D58A16B58]
2016-05-17 22:18:34,898 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event TASK_FINISHED to history file
2016-05-17 22:18:34,899 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1463523514422_0001_1][Event:VERTEX_FINISHED]: vertexName=F7D0FABC2C6A4B3B909C555D58A16B58, vertexId=vertex_1463523514422_0001_1_01, initRequestedTime=1463523514605, initedTime=1463523514621, startRequestedTime=1463523514642, startedTime=1463523514642, finishTime=1463523514898, timeTaken=256, status=SUCCEEDED, diagnostics=, counters=Counters: 46, File System Counters, FILE_BYTES_READ=354, FILE_BYTES_WRITTEN=32, FILE_READ_OPS=0, FILE_LARGE_READ_OPS=0, FILE_WRITE_OPS=0, org.apache.tez.common.counters.TaskCounter, REDUCE_INPUT_GROUPS=12, REDUCE_INPUT_RECORDS=5, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=13, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, GC_TIME_MILLIS=0, CPU_MILLISECONDS=0, PHYSICAL_MEMORY_BYTES=0, VIRTUAL_MEMORY_BYTES=0, COMMITTED_HEAP_BYTES=686292992, OUTPUT_RECORDS=5, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=322, SHUFFLE_BYTES=322, SHUFFLE_BYTES_DECOMPRESSED=318, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=322, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=5, MERGE_PHASE_TIME=9, FIRST_EVENT_RECEIVED=4, LAST_EVENT_RECEIVED=4, Shuffle Errors, BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0, cascading.flow.SliceCounters, Process_Begin_Time=1463523514887, Process_Duration=5, Process_End_Time=1463523514892, Read_Duration=1, Tuples_Read=5, Tuples_Written=5, Write_Duration=0, cascading.flow.StepCounters, Tuples_Written=5, vertexStats=firstTaskStartTime=1463523514852, firstTasksToStart=[ task_1463523514422_0001_1_01_000000 ], lastTaskFinishTime=1463523514896, lastTasksToFinish=[ task_1463523514422_0001_1_01_000000 ], minTaskDuration=44, maxTaskDuration=44, avgTaskDuration=44.0, numSuccessfulTasks=1, shortestDurationTasks=[ task_1463523514422_0001_1_01_000000 ], longestDurationTasks=[ task_1463523514422_0001_1_01_000000 ], vertexTaskStats={numFailedTasks=0, numSucceededTasks=1, numKilledTaskAttempts=0, numKilledTasks=0, numFailedTaskAttempts=0, numCompletedTasks=1}
2016-05-17 22:18:34,900 INFO impl.VertexImpl (VertexImpl.java:handle(1564)) - vertex_1463523514422_0001_1_01 [F7D0FABC2C6A4B3B909C555D58A16B58] transitioned from RUNNING to SUCCEEDED due to event V_TASK_COMPLETED
2016-05-17 22:18:34,900 INFO container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1463523514422_0001_00_000002 transitioned from RUNNING to IDLE via event C_TA_SUCCEEDED
2016-05-17 22:18:34,900 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1463523514422_0001_1][Event:CONTAINER_STOPPED]: containerId=container_1463523514422_0001_00_000002, stoppedTime=1463523514900, exitStatus=0
2016-05-17 22:18:34,900 INFO container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1463523514422_0001_00_000002 transitioned from IDLE to STOP_REQUESTED via event C_STOP_REQUEST
2016-05-17 22:18:34,900 INFO impl.DAGImpl (DAGImpl.java:transition(1747)) - Vertex vertex_1463523514422_0001_1_01 [F7D0FABC2C6A4B3B909C555D58A16B58] completed., numCompletedVertices=2, numSuccessfulVertices=2, numFailedVertices=0, numKilledVertices=0, numVertices=2
2016-05-17 22:18:34,900 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event VERTEX_FINISHED to history file
2016-05-17 22:18:34,900 INFO impl.DAGImpl (DAGImpl.java:commitOrAbortOutputs(831)) - Calling DAG commit/abort for dag: dag_1463523514422_0001_1
2016-05-17 22:18:34,901 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1463523514422_0001_1][Event:DAG_COMMIT_STARTED]: dagID=dag_1463523514422_0001_1
2016-05-17 22:18:34,901 INFO impl.DAGImpl (DAGImpl.java:commitOrAbortOutputs(871)) - No output committers for vertex: vertex_1463523514422_0001_1_00 [C982EAFB40CC492D8D2F1992C1E8AEF9]
2016-05-17 22:18:34,901 INFO impl.DAGImpl (DAGImpl.java:commitOrAbortOutputs(892)) - Committing output: 8C05B316B49B4068837D5FC66B1D2E75 for vertex: vertex_1463523514422_0001_1_01 [F7D0FABC2C6A4B3B909C555D58A16B58]
2016-05-17 22:18:34,901 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event CONTAINER_STOPPED to history file
2016-05-17 22:18:34,910 INFO app.TaskAttemptListenerImpTezDag (TaskAttemptListenerImpTezDag.java:getTask(213)) - Container with id: container_1463523514422_0001_00_000002 is valid, but no longer registered, and will be killed
2016-05-17 22:18:34,910 INFO task.ContainerReporter (ContainerReporter.java:call(65)) - Got TaskUpdate for containerId= container_1463523514422_0001_00_000002: 11 ms after starting to poll. TaskInfo: shouldDie: true
2016-05-17 22:18:34,910 INFO task.TezChild (TezChild.java:run(207)) - ContainerTask returned shouldDie=true for container container_1463523514422_0001_00_000002, Exiting
2016-05-17 22:18:34,910 INFO task.TezChild (TezChild.java:shutdown(343)) - Shutdown invoked for container container_1463523514422_0001_00_000002
2016-05-17 22:18:34,910 INFO task.TezChild (TezChild.java:shutdown(345)) - Shutting down container container_1463523514422_0001_00_000002
2016-05-17 22:18:34,910 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(270)) - ContainerExecutionResult for: container_1463523514422_0001_00_000002 = ContainerExecutionResult{exitStatus=SUCCESS, throwable=null, errorMessage='Asked to die by the AM'}
2016-05-17 22:18:34,911 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(274)) - Container: container_1463523514422_0001_00_000002 completed successfully
2016-05-17 22:18:34,912 INFO recovery.RecoveryService (RecoveryService.java:handle(306)) - DAG completed, dagId=dag_1463523514422_0001_1, queueSize=0
2016-05-17 22:18:34,912 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event DAG_FINISHED to history file
2016-05-17 22:18:34,913 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1463523514422_0001_1][Event:DAG_FINISHED]: dagId=dag_1463523514422_0001_1, startTime=1463523514596, finishTime=1463523514909, timeTaken=313, status=SUCCEEDED, diagnostics=, counters=Counters: 55, org.apache.tez.common.counters.DAGCounter, NUM_SUCCEEDED_TASKS=2, TOTAL_LAUNCHED_TASKS=2, RACK_LOCAL_TASKS=1, File System Counters, FILE_BYTES_READ=464, FILE_BYTES_WRITTEN=386, FILE_READ_OPS=0, FILE_LARGE_READ_OPS=0, FILE_WRITE_OPS=0, org.apache.tez.common.counters.TaskCounter, REDUCE_INPUT_GROUPS=12, REDUCE_INPUT_RECORDS=5, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=26, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, GC_TIME_MILLIS=111, CPU_MILLISECONDS=0, PHYSICAL_MEMORY_BYTES=0, VIRTUAL_MEMORY_BYTES=0, COMMITTED_HEAP_BYTES=1372585984, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=18, OUTPUT_BYTES=286, OUTPUT_BYTES_WITH_OVERHEAD=318, OUTPUT_BYTES_PHYSICAL=322, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=322, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_BYTES=322, SHUFFLE_BYTES_DECOMPRESSED=318, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=322, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=5, MERGE_PHASE_TIME=9, FIRST_EVENT_RECEIVED=4, LAST_EVENT_RECEIVED=4, Shuffle Errors, BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0, cascading.flow.SliceCounters, Process_Begin_Time=2927047029705, Process_Duration=8, Process_End_Time=2927047029713, Read_Duration=4, Tuples_Read=18, Tuples_Written=18, Write_Duration=0, cascading.flow.StepCounters, Tuples_Read=13, Tuples_Written=5
2016-05-17 22:18:34,913 INFO impl.DAGImpl (DAGImpl.java:finished(1180)) - DAG: dag_1463523514422_0001_1 finished with state: SUCCEEDED
2016-05-17 22:18:34,914 INFO impl.DAGImpl (DAGImpl.java:handle(973)) - dag_1463523514422_0001_1 transitioned from RUNNING to SUCCEEDED due to event DAG_VERTEX_COMPLETED
2016-05-17 22:18:34,915 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:stop(242)) - Ignoring stop request for containerId: container_1463523514422_0001_00_000002
2016-05-17 22:18:34,915 INFO container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1463523514422_0001_00_000002 transitioned from STOP_REQUESTED to COMPLETED via event C_COMPLETED
2016-05-17 22:18:34 Completed Dag: dag_1463523514422_0001_1
2016-05-17 22:18:34,917 INFO app.DAGAppMaster (DAGAppMaster.java:handle(579)) - DAG completed, dagId=dag_1463523514422_0001_1, dagState=SUCCEEDED
2016-05-17 22:18:34,917 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(164)) - Redirecting log file based on addend: dag_1463523514422_0001_1_post
2016-05-17 22:18:34,918 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(179)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-17 22:18:34,919 INFO app.DAGAppMaster (DAGAppMaster.java:handle(625)) - Waiting for next DAG to be submitted.
2016-05-17 22:18:34,923 INFO client.TezClient (TezClient.java:stop(495)) - Shutting down Tez Session, sessionName=(1/1) ...irstNBufferReverse/firstn, applicationId=application_1463523514422_0001
2016-05-17 22:18:34,926 INFO client.DAGClientHandler (DAGClientHandler.java:shutdownAM(117)) - Received message to shutdown AM
2016-05-17 22:18:34,927 INFO rm.TaskSchedulerEventHandler (TaskSchedulerEventHandler.java:setShouldUnregisterFlag(580)) - TaskScheduler notified that it should unregister from RM
2016-05-17 22:18:34,927 INFO app.DAGAppMaster (DAGAppMaster.java:shutdownTezAM(1088)) - No current running DAG, shutting down the AM
2016-05-17 22:18:34,927 INFO app.DAGAppMaster (DAGAppMaster.java:shutdown(692)) - DAGAppMasterShutdownHandler invoked
2016-05-17 22:18:34,927 INFO app.DAGAppMaster (DAGAppMaster.java:shutdown(701)) - Handling DAGAppMaster shutdown
2016-05-17 22:18:34,928 INFO app.DAGAppMaster (DAGAppMaster.java:run(721)) - Sleeping for 5 seconds before shutting down
2016-05-17 22:18:34,928 INFO history.HistoryEventHandler (HistoryEventHandler.java:serviceStop(79)) - Stopping HistoryEventHandler
2016-05-17 22:18:34,930 INFO recovery.RecoveryService (RecoveryService.java:serviceStop(189)) - Stopping RecoveryService
2016-05-17 22:18:34,930 INFO recovery.RecoveryService (RecoveryService.java:serviceStop(192)) - Handle the remaining events in queue, queue size=0
2016-05-17 22:18:34,931 INFO recovery.RecoveryService (RecoveryService.java:run(166)) - EventQueue take interrupted. Returning
2016-05-17 22:18:34,932 INFO recovery.RecoveryService (RecoveryService.java:serviceStop(214)) - Closing Summary Stream
2016-05-17 22:18:34,932 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:serviceStop(118)) - Stopping SimpleHistoryLoggingService, eventQueueBacklog=0
2016-05-17 22:18:34,932 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:run(105)) - EventQueue take interrupted. Returning
2016-05-17 22:18:34,938 INFO app.DAGAppMaster (DAGAppMaster.java:run(1773)) - Completed deletion of tez scratch data dir, path=file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.5.x-tez-0.6.x/build/test/tmp/.staging/C886F761D5D24A6BA33C0106FE67C18A/DB6F7C03C9E6433D8612EA31AE545BC1/.tez/application_1463523514422_0001
2016-05-17 22:18:34,939 INFO util.Hadoop18TapUtil (Hadoop18TapUtil.java:cleanTempPath(226)) - deleting temp path /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.5.x-tez-0.6.x/build/test/output/hadoop2-tez/bufferpipesplatform/testFirstNBufferReverse/firstn/_temporary
2016-05-17 22:18:34,940 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [test] completed in: 00:00.536
2016-05-17 22:18:35,018 INFO cascading.PlatformTestCase (PlatformTestCase.java:tearDown(189)) - copying to local /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.5.x-tez-0.6.x/build/test/output//hadoop2-tez/bufferpipesplatform/testFirstNBufferReverse/firstn
2016-05-17 22:18:35,043 INFO util.Util (Util.java:findMainClass(1374)) - using default application jar, may cause class not found exceptions on the cluster
2016-05-17 22:18:35,044 INFO planner.Hadoop2TezPlanner (Hadoop2TezPlanner.java:initialize(151)) - using application jar: /opt/concurrent/home/teamcity/.gradle/caches/modules-2/files-2.1/cascading/cascading-hadoop2-tez/3.1.0/226af02e24d5195c60e4ebe911d0c7c9780f59f0/cascading-hadoop2-tez-3.1.0.jar
2016-05-17 22:18:35,077 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [lower*upper] executed rule registry: NoHashJoinHadoop2TezRuleRegistry, completed as: ILLEGAL, in: 00:00.008
2016-05-17 22:18:35,081 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [lower*upper] executed rule registry: HashJoinHadoop2TezRuleRegistry, completed as: ILLEGAL, in: 00:00.011
2016-05-17 22:18:35,082 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [lower*upper] rule registry: NoHashJoinHadoop2TezRuleRegistry, found assembly to be malformed
2016-05-17 22:18:35,082 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [lower*upper] rule registry: HashJoinHadoop2TezRuleRegistry, found assembly to be malformed
2016-05-17 22:18:35,097 INFO app.DAGAppMaster (DAGAppMaster.java:run(731)) - Calling stop for all the services
2016-05-17 22:18:35,097 INFO app.DAGAppMaster (DAGAppMaster.java:run(742)) - Exiting DAGAppMaster..GoodBye!
2016-05-17 22:18:35,107 INFO util.Util (Util.java:findMainClass(1374)) - using default application jar, may cause class not found exceptions on the cluster
2016-05-17 22:18:35,107 INFO planner.Hadoop2TezPlanner (Hadoop2TezPlanner.java:initialize(151)) - using application jar: /opt/concurrent/home/teamcity/.gradle/caches/modules-2/files-2.1/cascading/cascading-hadoop2-tez/3.1.0/226af02e24d5195c60e4ebe911d0c7c9780f59f0/cascading-hadoop2-tez-3.1.0.jar
2016-05-17 22:18:35,147 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [test] executed rule registry: NoHashJoinHadoop2TezRuleRegistry, completed as: SUCCESS, in: 00:00.015
2016-05-17 22:18:35,164 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [test] executed rule registry: HashJoinHadoop2TezRuleRegistry, completed as: SUCCESS, in: 00:00.032
2016-05-17 22:18:35,165 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [test] rule registry: NoHashJoinHadoop2TezRuleRegistry, supports assembly with steps: 1, nodes: 2
2016-05-17 22:18:35,165 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [test] rule registry: HashJoinHadoop2TezRuleRegistry, supports assembly with steps: 1, nodes: 2
2016-05-17 22:18:35,165 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [test] rule registry: NoHashJoinHadoop2TezRuleRegistry, result was selected using: 'default comparator: selects plan with fewest steps and fewest nodes'
2016-05-17 22:18:35,188 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [test] tap forcing step to tez local mode: /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.5.x-tez-0.6.x/build/classes/test/data/lhs.txt
2016-05-17 22:18:35,199 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [test] tap forcing step to tez local mode: /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.5.x-tez-0.6.x/build/test/output/hadoop2-tez/bufferpipesplatform/testFirstNBuffer/firstn
2016-05-17 22:18:35,269 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [test] starting
2016-05-17 22:18:35,269 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [test] source: Hfs["TextDelimited[['num', 'lower']]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.5.x-tez-0.6.x/build/classes/test/data/lhs.txt"]
2016-05-17 22:18:35,270 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [test] sink: Hfs["TextDelimited[['num', 'lower']]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.5.x-tez-0.6.x/build/test/output/hadoop2-tez/bufferpipesplatform/testFirstNBuffer/firstn"]
2016-05-17 22:18:35,270 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [test] parallel execution of steps is enabled: false
2016-05-17 22:18:35,271 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [test] executing total steps: 1
2016-05-17 22:18:35,271 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [test] allocating management threads: 1
2016-05-17 22:18:35,272 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [test] starting step: (1/1) ...m/testFirstNBuffer/firstn
2016-05-17 22:18:35,273 WARN flow.Flow (BaseFlow.java:logWarn(1516)) - [test] 'yarn.timeline-service.enabled' is disabled, please enable to capture detailed metrics of completed flows, this may require starting the YARN timeline server daemon
2016-05-17 22:18:35,288 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [test] tez session mode enabled: true
2016-05-17 22:18:35,294 INFO client.TezClient (TezClient.java:<init>(153)) - Tez Client Version: [ component=tez-api, version=0.6.2, revision=c04283bf91fd96823eec37056d5c8b1a92728265, SCM-URL=scm:git:https://git-wip-us.apache.org/repos/asf/tez.git, buildTime=20150804-1436 ]
2016-05-17 22:18:35,295 INFO client.TezClient (TezClient.java:start(333)) - Session mode. Starting session.
2016-05-17 22:18:35,295 INFO client.TezClientUtils (TezClientUtils.java:setupTezJarsLocalResources(159)) - Ignoring 'tez.lib.uris' since 'tez.ignore.lib.uris' is set to true
2016-05-17 22:18:35,304 INFO client.TezClient (TezCommonUtils.java:createTezSystemStagingPath(118)) - Tez system stage directory file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.5.x-tez-0.6.x/build/test/tmp/.staging/B5FF3D6B3FEC4DE2963B32969B26C7D0/A34225503819433D86E0612345749768/.tez/application_1463523515295_0001 doesn't exist and is created
2016-05-17 22:18:35,321 INFO client.LocalClient (LocalClient.java:createDAGAppMaster(328)) - DAGAppMaster thread has been created
2016-05-17 22:18:35,322 INFO client.LocalClient (LocalClient.java:startDAGAppMaster(251)) - DAGAppMaster is not created wait for 100ms...
2016-05-17 22:18:35,322 INFO client.LocalClient (LocalClient.java:run(275)) - Using staging directory: /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.5.x-tez-0.6.x/build/test/tmp/.staging/B5FF3D6B3FEC4DE2963B32969B26C7D0/A34225503819433D86E0612345749768/.tez/application_1463523515295_0001
2016-05-17 22:18:35,325 INFO app.DAGAppMaster (DAGAppMaster.java:<init>(309)) - Created DAGAppMaster for application appattempt_1463523515295_0001_000000, versionInfo=[ component=tez-dag, version=0.6.2, revision=c04283bf91fd96823eec37056d5c8b1a92728265, SCM-URL=scm:git:https://git-wip-us.apache.org/repos/asf/tez.git, buildTime=20150804-1436 ]
2016-05-17 22:18:35,335 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(333)) - Comparing client version with AM version, clientVersion=0.6.2, AMVersion=0.6.2
2016-05-17 22:18:35,336 INFO app.DAGAppMaster (DAGAppMaster.java:serviceInit(386)) - Adding session token to jobTokenSecretManager for application
2016-05-17 22:18:35,337 INFO node.AMNodeTracker (AMNodeTracker.java:serviceInit(77)) - blacklistDisablePercent is 33, blacklistingEnabled: false, maxTaskFailuresPerNode: 1
2016-05-17 22:18:35,338 INFO history.HistoryEventHandler (HistoryEventHandler.java:serviceInit(50)) - Initializing HistoryEventHandler
2016-05-17 22:18:35,338 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:serviceInit(66)) - Log file location for SimpleHistoryLoggingService not specified, defaulting to containerLogDir=/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.5.x-tez-0.6.x/build/test/tmp/.staging/B5FF3D6B3FEC4DE2963B32969B26C7D0/A34225503819433D86E0612345749768/.tez/application_1463523515295_0001/localmode-log-dir
2016-05-17 22:18:35,338 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:serviceInit(88)) - Initializing SimpleHistoryLoggingService, logFileLocation=/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.5.x-tez-0.6.x/build/test/tmp/.staging/B5FF3D6B3FEC4DE2963B32969B26C7D0/A34225503819433D86E0612345749768/.tez/application_1463523515295_0001/localmode-log-dir/history.txt.appattempt_1463523515295_0001_000000, maxErrors=10
2016-05-17 22:18:35,338 INFO recovery.RecoveryService (RecoveryService.java:serviceInit(109)) - Initializing RecoveryService
2016-05-17 22:18:35,339 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:N/A][Event:AM_LAUNCHED]: appAttemptId=appattempt_1463523515295_0001_000000, appSubmitTime=1463523515325, launchTime=1463523515325
2016-05-17 22:18:35,346 INFO history.HistoryEventHandler (HistoryEventHandler.java:serviceStart(73)) - Starting HistoryEventHandler
2016-05-17 22:18:35,348 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:serviceStart(95)) - Starting SimpleHistoryLoggingService
2016-05-17 22:18:35,350 INFO client.DAGClientServer (DAGClientServer.java:serviceStart(91)) - Instantiated DAGClientRPCServer at ip-10-123-148-208.ec2.internal/10.123.148.208:48356
2016-05-17 22:18:35,351 INFO recovery.RecoveryService (RecoveryService.java:serviceStart(127)) - Starting RecoveryService
2016-05-17 22:18:35,351 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event AM_LAUNCHED to history file
2016-05-17 22:18:35,352 WARN impl.MetricsConfig (MetricsConfig.java:loadFirst(124)) - Cannot locate configuration: tried hadoop-metrics2-dagappmaster.properties,hadoop-metrics2.properties
2016-05-17 22:18:35,354 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:N/A][Event:AM_STARTED]: appAttemptId=appattempt_1463523515295_0001_000000, startTime=1463523515354
2016-05-17 22:18:35,355 INFO app.DAGAppMaster (DAGAppMaster.java:serviceStart(1644)) - In Session mode. Waiting for DAG over RPC
2016-05-17 22:18:35,354 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event AM_STARTED to history file
2016-05-17 22:18:35,422 INFO client.LocalClient (LocalClient.java:startDAGAppMaster(236)) - DAGAppMaster state: IDLE
2016-05-17 22:18:35,422 INFO client.TezClient (TezClient.java:start(367)) - The url to track the Tez Session: N/A
2016-05-17 22:18:35,422 INFO client.TezClient (TezClient.java:submitDAGSession(405)) - Submitting dag to TezSession, sessionName=(1/1) ...m/testFirstNBuffer/firstn, applicationId=application_1463523515295_0001, dagName=[B5FF3D6B3FEC4DE2963B32969B26C7D0/A34225503819433D86E0612345749768] test/(1/1) ...m/testFirstNBuffer/firstn
2016-05-17 22:18:35,438 INFO app.DAGAppMaster (DAGAppMaster.java:submitDAGToAppMaster(1126)) - Starting DAG submitted via RPC: [B5FF3D6B3FEC4DE2963B32969B26C7D0/A34225503819433D86E0612345749768] test/(1/1) ...m/testFirstNBuffer/firstn
2016-05-17 22:18:35,440 INFO app.DAGAppMaster (DAGAppMaster.java:generateDAGVizFile(870)) - Generating DAG graphviz file, dagId=dag_1463523515295_0001_1, filePath=/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.5.x-tez-0.6.x/build/test/tmp/.staging/B5FF3D6B3FEC4DE2963B32969B26C7D0/A34225503819433D86E0612345749768/.tez/application_1463523515295_0001/localmode-log-dir/dag_1463523515295_0001_1.dot
2016-05-17 22:18:35,441 INFO app.DAGAppMaster (DAGAppMaster.java:writePBTextFile(887)) - Writing DAG plan to: /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.5.x-tez-0.6.x/build/test/tmp/.staging/B5FF3D6B3FEC4DE2963B32969B26C7D0/A34225503819433D86E0612345749768/.tez/application_1463523515295_0001/localmode-log-dir/dag_1463523515295_0001_1-tez-dag.pb.txt
2016-05-17 22:18:35,455 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(164)) - Redirecting log file based on addend: dag_1463523515295_0001_1
2016-05-17 22:18:35,455 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(179)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-17 22:18:35,455 INFO app.DAGAppMaster (DAGAppMaster.java:startDAG(2051)) - Running DAG: [B5FF3D6B3FEC4DE2963B32969B26C7D0/A34225503819433D86E0612345749768] test/(1/1) ...m/testFirstNBuffer/firstn
2016-05-17 22:18:35 Running Dag: dag_1463523515295_0001_1
2016-05-17 22:18:35,468 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event DAG_SUBMITTED to history file
2016-05-17 22:18:35,469 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1463523515295_0001_1][Event:DAG_SUBMITTED]: dagID=dag_1463523515295_0001_1, submitTime=1463523515439
2016-05-17 22:18:35,470 INFO impl.VertexImpl (VertexImpl.java:setAdditionalOutputs(4033)) - setting additional outputs for vertex E045B238A1D94146A028712596348FCB
2016-05-17 22:18:35,470 INFO impl.DAGImpl (DAGImpl.java:assignDAGScheduler(1381)) - Using DAG Scheduler: org.apache.tez.dag.app.dag.impl.DAGSchedulerNaturalOrderControlled
2016-05-17 22:18:35,470 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event DAG_INITIALIZED to history file
2016-05-17 22:18:35,471 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1463523515295_0001_1][Event:DAG_INITIALIZED]: dagID=dag_1463523515295_0001_1, initTime=1463523515469
2016-05-17 22:18:35,471 INFO impl.DAGImpl (DAGImpl.java:handle(973)) - dag_1463523515295_0001_1 transitioned from NEW to INITED due to event DAG_INIT
2016-05-17 22:18:35,472 INFO client.TezClient (TezClient.java:submitDAGSession(478)) - Submitted dag to TezSession, sessionName=(1/1) ...m/testFirstNBuffer/firstn, applicationId=application_1463523515295_0001, dagName=[B5FF3D6B3FEC4DE2963B32969B26C7D0/A34225503819433D86E0612345749768] test/(1/1) ...m/testFirstNBuffer/firstn
2016-05-17 22:18:35,472 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1463523515295_0001_1][Event:DAG_STARTED]: dagID=dag_1463523515295_0001_1, startTime=1463523515472
2016-05-17 22:18:35,473 INFO impl.DAGImpl (DAGImpl.java:transition(1614)) - Added additional resources : [[]] to classpath
2016-05-17 22:18:35,473 INFO impl.DAGImpl (DAGImpl.java:handle(973)) - dag_1463523515295_0001_1 transitioned from INITED to RUNNING due to event DAG_START
2016-05-17 22:18:35,473 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event DAG_STARTED to history file
2016-05-17 22:18:35,473 INFO impl.VertexImpl (VertexImpl.java:setupVertex(2068)) - Root Inputs exist for Vertex: BC1BA212A5D84497A413594BBD27B056 : {29E4D85BDCF94FA3943A119D3FFB32DE={InputName=29E4D85BDCF94FA3943A119D3FFB32DE}, {Descriptor=ClassName=org.apache.tez.mapreduce.input.MRInput, hasPayload=true}, {ControllerDescriptor=ClassName=org.apache.tez.mapreduce.common.MRInputAMSplitGenerator, hasPayload=false}}
2016-05-17 22:18:35,473 INFO impl.VertexImpl (VertexImpl.java:setupVertex(2078)) - Starting root input initializer for input: 29E4D85BDCF94FA3943A119D3FFB32DE, with class: [org.apache.tez.mapreduce.common.MRInputAMSplitGenerator]
2016-05-17 22:18:35,473 INFO impl.VertexImpl (VertexImpl.java:assignVertexManager(2191)) - Setting vertexManager to RootInputVertexManager for vertex_1463523515295_0001_1_00 [BC1BA212A5D84497A413594BBD27B056]
2016-05-17 22:18:35,473 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(2983)) - Num tasks is -1. Expecting VertexManager/InputInitializers/1-1 split to set #tasks for the vertex vertex_1463523515295_0001_1_00 [BC1BA212A5D84497A413594BBD27B056]
2016-05-17 22:18:35,473 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(2987)) - Vertex will initialize from input initializer. vertex_1463523515295_0001_1_00 [BC1BA212A5D84497A413594BBD27B056]
2016-05-17 22:18:35,474 INFO impl.VertexImpl (VertexImpl.java:setupInputInitializerManager(3933)) - Vertex will initialize via inputInitializers vertex_1463523515295_0001_1_00 [BC1BA212A5D84497A413594BBD27B056]. Starting root input initializers: 1
2016-05-17 22:18:35,475 INFO impl.VertexImpl (VertexImpl.java:handle(1564)) - vertex_1463523515295_0001_1_00 [BC1BA212A5D84497A413594BBD27B056] transitioned from NEW to INITIALIZING due to event V_INIT
2016-05-17 22:18:35,475 INFO impl.VertexImpl (VertexImpl.java:assignVertexManager(2203)) - Setting vertexManager to ShuffleVertexManager for vertex_1463523515295_0001_1_01 [E045B238A1D94146A028712596348FCB]
2016-05-17 22:18:35,475 INFO dag.RootInputInitializerManager (RootInputInitializerManager.java:run(242)) - Starting InputInitializer for Input: 29E4D85BDCF94FA3943A119D3FFB32DE on vertex vertex_1463523515295_0001_1_00 [BC1BA212A5D84497A413594BBD27B056]
2016-05-17 22:18:35,478 INFO common.MRInputAMSplitGenerator (MRInputAMSplitGenerator.java:initialize(92)) - Emitting serialized splits: true
2016-05-17 22:18:35,479 INFO common.MRInputAMSplitGenerator (MRInputAMSplitGenerator.java:initialize(110)) - Input 29E4D85BDCF94FA3943A119D3FFB32DE asking for 1 tasks. Headroom: 672 Task Resource: 1024 waves: 1.7
2016-05-17 22:18:35,478 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:initialize(696)) - Shuffle Vertex Manager: settings minFrac:0.25 maxFrac:0.75 auto:false desiredTaskIput:104857600 minTasks:1
2016-05-17 22:18:35,480 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(3014)) - Creating 1 tasks for vertex: vertex_1463523515295_0001_1_01 [E045B238A1D94146A028712596348FCB]
2016-05-17 22:18:35,480 INFO impl.VertexImpl (VertexImpl.java:handleInitEvent(3026)) - Directly initializing vertex: vertex_1463523515295_0001_1_01 [E045B238A1D94146A028712596348FCB]
2016-05-17 22:18:35,481 INFO impl.VertexImpl (VertexImpl.java:initializeCommitters(1915)) - Invoking committer inits for vertex, vertexId=vertex_1463523515295_0001_1_01 [E045B238A1D94146A028712596348FCB]
2016-05-17 22:18:35,481 INFO impl.VertexImpl (VertexImpl.java:initializeCommitters(1927)) - Instantiating committer for output=26F4A514C34C40A88954A24ACAF1AF82, vertexId=vertex_1463523515295_0001_1_01 [E045B238A1D94146A028712596348FCB], committerClass=org.apache.tez.mapreduce.committer.MROutputCommitter
2016-05-17 22:18:35,482 INFO impl.VertexImpl (VertexImpl.java:run(1945)) - Invoking committer init for output=26F4A514C34C40A88954A24ACAF1AF82, vertexId=vertex_1463523515295_0001_1_01 [E045B238A1D94146A028712596348FCB]
2016-05-17 22:18:35,483 INFO common.MRInputAMSplitGenerator (MRInputAMSplitGenerator.java:initialize(121)) - Grouping input splits
2016-05-17 22:18:35,484 INFO hadoop.MRInputHelpers (MRInputHelpers.java:generateInputSplitsToMem(294)) - Generating mapred api input splits
2016-05-17 22:18:35,486 INFO hadoop.MRInputHelpers (MRInputHelpers.java:generateInputSplitsToMem(300)) - NumSplits: 1, SerializedSize: 396
2016-05-17 22:18:35,486 INFO dag.RootInputInitializerManager (RootInputInitializerManager.java:onSuccess(272)) - Succeeded InputInitializer for Input: 29E4D85BDCF94FA3943A119D3FFB32DE on vertex vertex_1463523515295_0001_1_00 [BC1BA212A5D84497A413594BBD27B056]
2016-05-17 22:18:35,489 INFO committer.MROutputCommitter (MROutputCommitter.java:getOutputCommitter(145)) - OutputCommitter for outputName=26F4A514C34C40A88954A24ACAF1AF82, vertexName=E045B238A1D94146A028712596348FCB, outputCommitterClass=org.apache.hadoop.mapred.FileOutputCommitter
2016-05-17 22:18:35,489 INFO impl.VertexImpl (VertexImpl.java:run(1949)) - Invoking committer setup for output=26F4A514C34C40A88954A24ACAF1AF82, vertexId=vertex_1463523515295_0001_1_01 [E045B238A1D94146A028712596348FCB]
2016-05-17 22:18:35,492 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event VERTEX_INITIALIZED to history file
2016-05-17 22:18:35,493 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1463523515295_0001_1][Event:VERTEX_INITIALIZED]: vertexName=E045B238A1D94146A028712596348FCB, vertexId=vertex_1463523515295_0001_1_01, initRequestedTime=1463523515475, initedTime=1463523515492, numTasks=1, processorName=cascading.flow.tez.FlowProcessor, additionalInputsCount=0
2016-05-17 22:18:35,493 INFO impl.VertexImpl (VertexImpl.java:handle(1564)) - vertex_1463523515295_0001_1_01 [E045B238A1D94146A028712596348FCB] transitioned from NEW to INITED due to event V_INIT
2016-05-17 22:18:35,493 INFO impl.VertexImpl (VertexImpl.java:canInitVertex(3195)) - Cannot init vertex: vertex_1463523515295_0001_1_00 [BC1BA212A5D84497A413594BBD27B056] numTasks: -1 numUnitializedEdges: 0 numInitializedInputs: 0 initWaitsForRootInitializers: true
2016-05-17 22:18:35,494 INFO impl.VertexImpl (VertexImpl.java:setParallelism(1374)) - Got updated RootInputsSpecs: {29E4D85BDCF94FA3943A119D3FFB32DE=forAllWorkUnits=true, update=[1]}
2016-05-17 22:18:35,494 INFO impl.VertexImpl (VertexImpl.java:setParallelism(1393)) - Vertex vertex_1463523515295_0001_1_00 [BC1BA212A5D84497A413594BBD27B056] parallelism set to 1
2016-05-17 22:18:35,495 INFO impl.VertexImpl (VertexImpl.java:canInitVertex(3195)) - Cannot init vertex: vertex_1463523515295_0001_1_00 [BC1BA212A5D84497A413594BBD27B056] numTasks: 1 numUnitializedEdges: 0 numInitializedInputs: 0 initWaitsForRootInitializers: true
2016-05-17 22:18:35,495 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [test] submitted tez dag to app master: application_1463523515295_0001, with dag id: dag_1463523515295_0001_1
2016-05-17 22:18:35,499 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1463523515295_0001_1][Event:VERTEX_DATA_MOVEMENT_EVENTS_GENERATED]: vertexId=vertex_1463523515295_0001_1_00, eventCount=1
2016-05-17 22:18:35,499 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event VERTEX_INITIALIZED to history file
2016-05-17 22:18:35,499 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1463523515295_0001_1][Event:VERTEX_INITIALIZED]: vertexName=BC1BA212A5D84497A413594BBD27B056, vertexId=vertex_1463523515295_0001_1_00, initRequestedTime=1463523515473, initedTime=1463523515499, numTasks=1, processorName=cascading.flow.tez.FlowProcessor, additionalInputsCount=1
2016-05-17 22:18:35,500 INFO impl.VertexImpl (VertexImpl.java:startIfPossible(3061)) - Triggering start event for vertex: vertex_1463523515295_0001_1_00 [BC1BA212A5D84497A413594BBD27B056] with distanceFromRoot: 0
2016-05-17 22:18:35,500 INFO impl.VertexImpl (VertexImpl.java:handle(1564)) - vertex_1463523515295_0001_1_00 [BC1BA212A5D84497A413594BBD27B056] transitioned from INITIALIZING to INITED due to event V_ROOT_INPUT_INITIALIZED
2016-05-17 22:18:35,500 INFO impl.ImmediateStartVertexManager (ImmediateStartVertexManager.java:scheduleTasks(99)) - Starting 1 in BC1BA212A5D84497A413594BBD27B056
2016-05-17 22:18:35,501 INFO impl.VertexImpl (VertexImpl.java:scheduleTasks(1250)) - Routing pending task events for vertex: vertex_1463523515295_0001_1_00 [BC1BA212A5D84497A413594BBD27B056]
2016-05-17 22:18:35,501 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event VERTEX_STARTED to history file
2016-05-17 22:18:35,501 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1463523515295_0001_1][Event:VERTEX_STARTED]: vertexId=vertex_1463523515295_0001_1_00, startRequestedTime=1463523515500, startedTime=1463523515500
2016-05-17 22:18:35,502 INFO impl.VertexImpl (VertexImpl.java:handle(1564)) - vertex_1463523515295_0001_1_00 [BC1BA212A5D84497A413594BBD27B056] transitioned from INITED to RUNNING due to event V_START
2016-05-17 22:18:35,502 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event TASK_STARTED to history file
2016-05-17 22:18:35,503 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1463523515295_0001_1][Event:TASK_STARTED]: vertexName=BC1BA212A5D84497A413594BBD27B056, taskId=task_1463523515295_0001_1_00_000000, scheduledTime=1463523515502, launchTime=1463523515502
2016-05-17 22:18:35,503 INFO impl.TaskImpl (TaskImpl.java:handle(862)) - task_1463523515295_0001_1_00_000000 Task Transitioned from NEW to SCHEDULED due to event T_SCHEDULE
2016-05-17 22:18:35,503 INFO impl.VertexImpl (VertexImpl.java:transition(3155)) - Source vertex started: vertex_1463523515295_0001_1_00 for vertex: vertex_1463523515295_0001_1_01 [E045B238A1D94146A028712596348FCB] numStartedSources: 1 numSources: 1
2016-05-17 22:18:35,503 INFO impl.VertexImpl (VertexImpl.java:startIfPossible(3061)) - Triggering start event for vertex: vertex_1463523515295_0001_1_01 [E045B238A1D94146A028712596348FCB] with distanceFromRoot: 1
2016-05-17 22:18:35,505 INFO impl.DAGSchedulerNaturalOrderControlled (DAGSchedulerNaturalOrderControlled.java:scheduleTask(111)) - Scheduled vertex: vertex_1463523515295_0001_1_00 [BC1BA212A5D84497A413594BBD27B056]
2016-05-17 22:18:35,505 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:handleVertexStateUpdate(720)) - Received configured notification : CONFIGURED for vertex: BC1BA212A5D84497A413594BBD27B056 in vertex: E045B238A1D94146A028712596348FCB
2016-05-17 22:18:35,506 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:onVertexStarted(355)) - OnVertexStarted vertex: E045B238A1D94146A028712596348FCB with 1 source tasks and 1 pending tasks
2016-05-17 22:18:35,506 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1463523515295_0001_1][Event:VERTEX_STARTED]: vertexId=vertex_1463523515295_0001_1_01, startRequestedTime=1463523515505, startedTime=1463523515505
2016-05-17 22:18:35,506 INFO impl.VertexImpl (VertexImpl.java:handle(1564)) - vertex_1463523515295_0001_1_01 [E045B238A1D94146A028712596348FCB] transitioned from INITED to RUNNING due to event V_START
2016-05-17 22:18:35,506 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event VERTEX_STARTED to history file
2016-05-17 22:18:35,507 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:transition(1014)) - remoteTaskSpec:DAGName : [B5FF3D6B3FEC4DE2963B32969B26C7D0/A34225503819433D86E0612345749768] test/(1/1) ...m/testFirstNBuffer/firstn, VertexName: BC1BA212A5D84497A413594BBD27B056, VertexParallelism: 1, TaskAttemptID:attempt_1463523515295_0001_1_00_000000_0, processorName=cascading.flow.tez.FlowProcessor, inputSpecListSize=1, outputSpecListSize=1, inputSpecList=[{{ sourceVertexName=29E4D85BDCF94FA3943A119D3FFB32DE, physicalEdgeCount=1, inputClassName=org.apache.tez.mapreduce.input.MRInput }}, ], outputSpecList=[{{ destinationVertexName=E045B238A1D94146A028712596348FCB, physicalEdgeCount=1, outputClassName=org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput }}, ]
2016-05-17 22:18:35,507 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:handle(692)) - attempt_1463523515295_0001_1_00_000000_0 TaskAttempt Transitioned from NEW to START_WAIT due to event TA_SCHEDULE
2016-05-17 22:18:35,507 INFO rm.TaskSchedulerEventHandler (TaskSchedulerEventHandler.java:handleEvent(149)) - Processing the event EventType: S_TA_LAUNCH_REQUEST
2016-05-17 22:18:35,509 INFO node.AMNodeTracker (AMNodeTracker.java:nodeSeen(91)) - Adding new node: 127.0.0.1:0
2016-05-17 22:18:35,509 INFO container.AMContainerHelpers (AMContainerHelpers.java:createCommonContainerLaunchContext(112)) - Adding #1 tokens and #0 secret keys for NM use for launching container
2016-05-17 22:18:35,509 INFO container.AMContainerHelpers (AMContainerHelpers.java:createCommonContainerLaunchContext(122)) - Putting shuffle token in serviceData
2016-05-17 22:18:35,510 INFO container.AMContainerImpl (AMContainerImpl.java:transition(521)) - Sending Launch Request for Container with id: container_1463523515295_0001_00_000001
2016-05-17 22:18:35,510 INFO container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1463523515295_0001_00_000001 transitioned from ALLOCATED to LAUNCHING via event C_LAUNCH_REQUEST
2016-05-17 22:18:35,512 INFO task.TezChild (TezChild.java:newTezChild(432)) - PID, containerIdentifier: null, container_1463523515295_0001_00_000001
2016-05-17 22:18:35,512 WARN impl.MetricsSystemImpl (MetricsSystemImpl.java:init(145)) - DAGAppMaster metrics system already initialized!
2016-05-17 22:18:35,513 INFO container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1463523515295_0001_00_000001 transitioned from LAUNCHING to IDLE via event C_LAUNCHED
2016-05-17 22:18:35,513 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1463523515295_0001_1][Event:CONTAINER_LAUNCHED]: containerId=container_1463523515295_0001_00_000001, launchTime=1463523515513
2016-05-17 22:18:35,513 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event CONTAINER_LAUNCHED to history file
2016-05-17 22:18:35,514 INFO task.ContainerReporter (ContainerReporter.java:call(55)) - Attempting to fetch new task for container container_1463523515295_0001_00_000001
2016-05-17 22:18:35,515 INFO container.AMContainerImpl (AMContainerImpl.java:transition(827)) - Assigned taskAttempt + [attempt_1463523515295_0001_1_00_000000_0] to container: [container_1463523515295_0001_00_000001]
2016-05-17 22:18:35,515 INFO container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1463523515295_0001_00_000001 transitioned from IDLE to RUNNING via event C_PULL_TA
2016-05-17 22:18:35,515 INFO app.TaskAttemptListenerImpTezDag (TaskAttemptListenerImpTezDag.java:getTask(240)) - Container with id: container_1463523515295_0001_00_000001 given task: attempt_1463523515295_0001_1_00_000000_0
2016-05-17 22:18:35,515 INFO task.ContainerReporter (ContainerReporter.java:call(65)) - Got TaskUpdate for containerId= container_1463523515295_0001_00_000001: 0 ms after starting to poll. TaskInfo: shouldDie: false, currentTaskAttemptId: attempt_1463523515295_0001_1_00_000000_0
2016-05-17 22:18:35,516 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:transition(1146)) - TaskAttempt: [attempt_1463523515295_0001_1_00_000000_0] started. Is using containerId: [container_1463523515295_0001_00_000001] on NM: [127.0.0.1:0]
2016-05-17 22:18:35 Starting to run new task attempt: attempt_1463523515295_0001_1_00_000000_0
2016-05-17 22:18:35,516 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(164)) - Redirecting log file based on addend: attempt_1463523515295_0001_1_00_000000_0
2016-05-17 22:18:35,516 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(179)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-17 22:18:35,516 INFO task.TezChild (TezChild.java:handleNewTaskCredentials(273)) - Refreshing UGI since Credentials have changed
2016-05-17 22:18:35,517 INFO task.TezChild (TezChild.java:handleNewTaskCredentials(276)) - Credentials : #Tokens=1, #SecretKeys=0
2016-05-17 22:18:35,516 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1463523515295_0001_1][Event:TASK_ATTEMPT_STARTED]: vertexName=BC1BA212A5D84497A413594BBD27B056, taskAttemptId=attempt_1463523515295_0001_1_00_000000_0, startTime=1463523515516, containerId=container_1463523515295_0001_00_000001, nodeId=127.0.0.1:0, inProgressLogs=127.0.0.1:0/node/containerlogs/container_1463523515295_0001_00_000001/teamcity, completedLogs=
2016-05-17 22:18:35,517 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:handle(692)) - attempt_1463523515295_0001_1_00_000000_0 TaskAttempt Transitioned from START_WAIT to RUNNING due to event TA_STARTED_REMOTELY
2016-05-17 22:18:35,516 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event TASK_ATTEMPT_STARTED to history file
2016-05-17 22:18:35,517 INFO impl.TaskImpl (TaskImpl.java:handle(862)) - task_1463523515295_0001_1_00_000000 Task Transitioned from SCHEDULED to RUNNING due to event T_ATTEMPT_LAUNCHED
2016-05-17 22:18:35,517 INFO task.TezChild (TezChild.java:handleNewTaskLocalResources(302)) - Localizing additional local resources for Task : {}
2016-05-17 22:18:35,518 INFO task.TezChild (TezChild.java:handleNewTaskLocalResources(312)) - Done localizing additional resources
2016-05-17 22:18:35,518 INFO metrics.TaskCounterUpdater (TaskCounterUpdater.java:initResourceCalculatorPlugin(148)) - Using ResourceCalculatorProcessTree : [ ]
2016-05-17 22:18:35,519 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:<init>(149)) - Initializing LogicalIOProcessorRuntimeTask with TaskSpec: DAGName : [B5FF3D6B3FEC4DE2963B32969B26C7D0/A34225503819433D86E0612345749768] test/(1/1) ...m/testFirstNBuffer/firstn, VertexName: BC1BA212A5D84497A413594BBD27B056, VertexParallelism: 1, TaskAttemptID:attempt_1463523515295_0001_1_00_000000_0, processorName=cascading.flow.tez.FlowProcessor, inputSpecListSize=1, outputSpecListSize=1, inputSpecList=[{{ sourceVertexName=29E4D85BDCF94FA3943A119D3FFB32DE, physicalEdgeCount=1, inputClassName=org.apache.tez.mapreduce.input.MRInput }}, ], outputSpecList=[{{ destinationVertexName=E045B238A1D94146A028712596348FCB, physicalEdgeCount=1, outputClassName=org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput }}, ]
2016-05-17 22:18:35,519 INFO resources.MemoryDistributor (MemoryDistributor.java:<init>(85)) - InitialMemoryDistributor (isEnabled=true) invoked with: numInputs=1, numOutputs=1, JVM.maxFree=704643072
2016-05-17 22:18:35,520 INFO task.TezTaskRunner (TezTaskRunner.java:run(172)) - Initializing task, taskAttemptId=attempt_1463523515295_0001_1_00_000000_0
2016-05-17 22:18:35,520 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(191)) - Initializing LogicalProcessorIORuntimeTask
2016-05-17 22:18:35,520 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(195)) - Creating processor, processorClassName=cascading.flow.tez.FlowProcessor
2016-05-17 22:18:35,521 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(389)) - Initializing Input using InputSpec: { sourceVertexName=29E4D85BDCF94FA3943A119D3FFB32DE, physicalEdgeCount=1, inputClassName=org.apache.tez.mapreduce.input.MRInput }
2016-05-17 22:18:35,521 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initializeLogicalIOProcessor(488)) - Initializing processor, processorClassName=cascading.flow.tez.FlowProcessor
2016-05-17 22:18:35,521 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(438)) - Initializing Output using OutputSpec: { destinationVertexName=E045B238A1D94146A028712596348FCB, physicalEdgeCount=1, outputClassName=org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput }
2016-05-17 22:18:35,522 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:createInput(537)) - Creating Input
2016-05-17 22:18:35,522 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(397)) - Initializing Input with src edge: 29E4D85BDCF94FA3943A119D3FFB32DE
2016-05-17 22:18:35,522 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:createOutput(565)) - Creating Output
2016-05-17 22:18:35,522 INFO resources.MemoryDistributor$RequestorInfo (MemoryDistributor.java:<init>(233)) - Received request: 0, type: INPUT, componentVertexName: 29E4D85BDCF94FA3943A119D3FFB32DE
2016-05-17 22:18:35,522 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(446)) - Initializing Output with dest edge: E045B238A1D94146A028712596348FCB
2016-05-17 22:18:35,526 INFO tez.FlowProcessor (FlowProcessor.java:initialize(88)) - cascading version: 3.1.0
2016-05-17 22:18:35,526 INFO impl.ExternalSorter (ExternalSorter.java:getInitialMemoryRequirement(293)) - Requested SortBufferSize (tez.runtime.io.sort.mb): 100
2016-05-17 22:18:35,526 INFO resources.MemoryDistributor$RequestorInfo (MemoryDistributor.java:<init>(233)) - Received request: 104857600, type: OUTPUT, componentVertexName: E045B238A1D94146A028712596348FCB
2016-05-17 22:18:35,638 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(452)) - Initialized Output with dest edge: E045B238A1D94146A028712596348FCB
2016-05-17 22:18:35,639 INFO impl.TaskImpl (TaskImpl.java:getTaskAttemptTezEvents(481)) - TaskAttempt:attempt_1463523515295_0001_1_00_000000_0 sent events: (0-1)
2016-05-17 22:18:35,639 INFO input.MRInput (MRInput.java:initialize(440)) - Using New mapreduce API: false, split information via event: true
2016-05-17 22:18:35,640 INFO input.MRInput (MRInput.java:initializeInternal(493)) - Initialzed MRInput: 29E4D85BDCF94FA3943A119D3FFB32DE
2016-05-17 22:18:35,640 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(403)) - Initialized Input with src edge: 29E4D85BDCF94FA3943A119D3FFB32DE
2016-05-17 22:18:35,643 INFO tez.FlowProcessor (FlowProcessor.java:initialize(94)) - flow node id: BC1BA212A5D84497A413594BBD27B056, ordinal: 0
2016-05-17 22:18:35,643 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: BC1BA212A5D84497A413594BBD27B056, mem on start (mb), free: 687, total: 713, max: 713
2016-05-17 22:18:35,643 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initializeLogicalIOProcessor(491)) - Initialized processor, processorClassName=cascading.flow.tez.FlowProcessor
2016-05-17 22:18:35,643 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(220)) - Waiting for 2 initializers to finish
2016-05-17 22:18:35,643 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(220)) - Waiting for 1 initializers to finish
2016-05-17 22:18:35,643 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(233)) - All initializers finished
2016-05-17 22:18:35,644 INFO resources.MemoryDistributor (MemoryDistributor.java:makeInitialAllocations(124)) - Using Allocator class: org.apache.tez.runtime.library.resources.WeightedScalingMemoryDistributor
2016-05-17 22:18:35,644 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:getRequestTypeForClass(197)) - Falling back to RequestType.OTHER for class: org.apache.tez.mapreduce.input.MRInput
2016-05-17 22:18:35,644 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:initialProcessMemoryRequestContext(171)) - ScaleFactor: 1, for type: OTHER
2016-05-17 22:18:35,645 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:initialProcessMemoryRequestContext(171)) - ScaleFactor: 12, for type: SORTED_OUTPUT
2016-05-17 22:18:35,645 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:computeReservedFraction(262)) - InitialReservationFraction=0.3, AdditionalReservationFractionForIOs=0.03, finalReserveFractionUsed=0.32999999999999996
2016-05-17 22:18:35,645 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:assignMemory(126)) - Scaling Requests. NumRequests: 2, numScaledRequests: 13, TotalRequested: 104857600, TotalRequestedScaled: 9.679163076923077E7, TotalJVMHeap: 747634688, TotalAvailable: 472110858, TotalRequested/TotalJVMHeap:0.14
2016-05-17 22:18:35,646 INFO resources.MemoryDistributor (MemoryDistributor.java:makeInitialAllocations(138)) - Informing: INPUT, 29E4D85BDCF94FA3943A119D3FFB32DE, org.apache.tez.mapreduce.input.MRInput: requested=0, allocated=0
2016-05-17 22:18:35,646 INFO resources.MemoryDistributor (MemoryDistributor.java:makeInitialAllocations(138)) - Informing: OUTPUT, E045B238A1D94146A028712596348FCB, org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput: requested=104857600, allocated=104857600
2016-05-17 22:18:35,646 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(255)) - Starting Inputs/Outputs
2016-05-17 22:18:35,646 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(268)) - Input: 29E4D85BDCF94FA3943A119D3FFB32DE being auto started by the framework. Subsequent instances will not be auto-started
2016-05-17 22:18:35,646 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(419)) - Starting Input with src edge: 29E4D85BDCF94FA3943A119D3FFB32DE
2016-05-17 22:18:35,646 INFO input.MRInput (MRInput.java:start(450)) - MRInput setup to received events1
2016-05-17 22:18:35,647 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(421)) - Started Input with src edge: 29E4D85BDCF94FA3943A119D3FFB32DE
2016-05-17 22:18:35,646 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(289)) - Num IOs determined for AutoStart: 1
2016-05-17 22:18:35,647 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(291)) - Waiting for 1 IOs to start
2016-05-17 22:18:35,647 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(304)) - AutoStartComplete
2016-05-17 22:18:35,648 INFO task.TezTaskRunner (TezTaskRunner.java:run(175)) - Running task, taskAttemptId=attempt_1463523515295_0001_1_00_000000_0
2016-05-17 22:18:35,648 INFO input.MRInput (MRInput.java:initFromEventInternal(619)) - Initializing RecordReader from event
2016-05-17 22:18:35,648 INFO input.MRInput (MRInput.java:initFromEventInternal(630)) - Split Details -> SplitClass: org.apache.hadoop.mapred.split.TezGroupedSplit, OldSplit: org.apache.hadoop.mapred.split.TezGroupedSplit@781d8448
2016-05-17 22:18:35,650 INFO lib.MRReaderMapred (MRReaderMapred.java:setIncrementalConfigParams(163)) - Processing split: org.apache.hadoop.mapred.split.TezGroupedSplit@781d8448
2016-05-17 22:18:35,650 INFO input.MRInput (MRInput.java:initFromEventInternal(634)) - Initialized RecordReader from event
2016-05-17 22:18:35,650 INFO input.MRInput (MRInput.java:processSplitEvent(586)) - Notifying on RecordReader Initialized
2016-05-17 22:18:35,651 INFO graph.Hadoop2TezStreamGraph (Hadoop2TezStreamGraph.java:buildGraph(151)) - using streamed source: Hfs["TextDelimited[['num', 'lower']]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.5.x-tez-0.6.x/build/classes/test/data/lhs.txt"]
2016-05-17 22:18:35,651 INFO tez.FlowProcessor (FlowProcessor.java:run(121)) - sourcing from: Hfs["TextDelimited[['num', 'lower']]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.5.x-tez-0.6.x/build/classes/test/data/lhs.txt"] streamed: true, id: 29E4D85BDCF94FA3943A119D3FFB32DE
2016-05-17 22:18:35,652 INFO tez.FlowProcessor (FlowProcessor.java:run(124)) - sinking to: GroupBy(test)[by:[{1}:'num']], id: CE3CB083A5F44E699FADD0A5DDB4937D
2016-05-17 22:18:35,652 INFO element.TezGroupGate (TezGroupGate.java:prepare(98)) - calling OrderedPartitionedKVOutput#start() on: GroupBy(test)[by:[{1}:'num']] CE3CB083A5F44E699FADD0A5DDB4937D
2016-05-17 22:18:35,655 INFO impl.ExternalSorter (ExternalSorter.java:<init>(167)) - keySerializer=cascading.tuple.hadoop.io.KeyTupleSerializer@2e119953; valueSerializer=cascading.tuple.hadoop.io.ValueTupleSerializer@5692fc6; comparator=cascading.tuple.hadoop.util.TupleComparator@3cbf72f0; conf=cascading.tuple.hadoop.TupleSerialization,org.apache.hadoop.io.serializer.WritableSerialization,org.apache.hadoop.io.serializer.avro.AvroSpecificSerialization,org.apache.hadoop.io.serializer.avro.AvroReflectSerialization
2016-05-17 22:18:35,655 INFO impl.ExternalSorter (ExternalSorter.java:<init>(224)) - Instantiating Partitioner: [cascading.tuple.tez.util.TuplePartitioner]
2016-05-17 22:18:35,655 INFO common.TezRuntimeUtils (TezRuntimeUtils.java:instantiatePartitioner(109)) - Using partitioner class: cascading.tuple.tez.util.TuplePartitioner
2016-05-17 22:18:35,656 INFO common.TezRuntimeUtils (TezRuntimeUtils.java:instantiateCombiner(65)) - No combiner specified via tez.runtime.combiner.class. Combiner will not be used
2016-05-17 22:18:35,670 INFO dflt.DefaultSorter (DefaultSorter.java:setEquator(328)) - (EQUATOR) 0 kvi 26214396(104857584)
2016-05-17 22:18:35,670 INFO dflt.DefaultSorter (DefaultSorter.java:<init>(150)) - tez.runtime.io.sort.mb: 100
2016-05-17 22:18:35,670 INFO dflt.DefaultSorter (DefaultSorter.java:<init>(151)) - soft limit at 83886080
2016-05-17 22:18:35,670 INFO dflt.DefaultSorter (DefaultSorter.java:<init>(152)) - bufstart = 0; bufvoid = 104857600
2016-05-17 22:18:35,670 INFO dflt.DefaultSorter (DefaultSorter.java:<init>(153)) - kvstart = 26214396; length = 6553600
2016-05-17 22:18:35,671 INFO element.TezSourceStage (TezSourceStage.java:prepare(63)) - calling MRInput#start() on: Hfs["TextDelimited[['num', 'lower']]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.5.x-tez-0.6.x/build/classes/test/data/lhs.txt"]
2016-05-17 22:18:35,671 INFO input.MRInput (MRInput.java:start(450)) - MRInput setup to received events1
2016-05-17 22:18:35,672 INFO tez.FlowProcessor (FlowProcessor.java:waitForInputsReady(202)) - flow node id: BC1BA212A5D84497A413594BBD27B056, all 1 inputs ready in: 00:00:00.000
2016-05-17 22:18:35,675 INFO task.TezTaskRunner (TezTaskRunner.java:run(177)) - Closing task, taskAttemptId=attempt_1463523515295_0001_1_00_000000_0
2016-05-17 22:18:35,675 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: BC1BA212A5D84497A413594BBD27B056, mem on close (mb), free: 585, total: 713, max: 713
2016-05-17 22:18:35,675 INFO dflt.DefaultSorter (DefaultSorter.java:flush(589)) - Starting flush of map output
2016-05-17 22:18:35,675 INFO dflt.DefaultSorter (DefaultSorter.java:flush(607)) - Sorting & Spilling map output
2016-05-17 22:18:35,676 INFO dflt.DefaultSorter (DefaultSorter.java:flush(608)) - bufstart = 0; bufend = 195; bufvoid = 104857600
2016-05-17 22:18:35,676 INFO dflt.DefaultSorter (DefaultSorter.java:flush(610)) - kvstart = 26214396(104857584); kvend = 26214348(104857392); length = 49/6553600
2016-05-17 22:18:35,680 INFO dflt.DefaultSorter (DefaultSorter.java:spill(829)) - Finished spill 0
2016-05-17 22:18:35 Completed running task attempt: attempt_1463523515295_0001_1_00_000000_0
2016-05-17 22:18:35,685 INFO task.TezTaskRunner (TezTaskRunner.java:run(181)) - Task completed, taskAttemptId=attempt_1463523515295_0001_1_00_000000_0, fatalErrorOccurred=false
2016-05-17 22:18:35,686 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1463523515295_0001_1][Event:VERTEX_DATA_MOVEMENT_EVENTS_GENERATED]: vertexId=vertex_1463523515295_0001_1_00, eventCount=1
2016-05-17 22:18:35,686 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event TASK_ATTEMPT_FINISHED to history file
2016-05-17 22:18:35,686 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1463523515295_0001_1][Event:TASK_ATTEMPT_FINISHED]: vertexName=BC1BA212A5D84497A413594BBD27B056, taskAttemptId=attempt_1463523515295_0001_1_00_000000_0, startTime=1463523515516, finishTime=1463523515686, timeTaken=170, status=SUCCEEDED, errorEnum=, diagnostics=, counters=Counters: 28, org.apache.tez.common.counters.DAGCounter, RACK_LOCAL_TASKS=1, File System Counters, FILE_BYTES_READ=110, FILE_BYTES_WRITTEN=727, FILE_READ_OPS=0, FILE_LARGE_READ_OPS=0, FILE_WRITE_OPS=0, org.apache.tez.common.counters.TaskCounter, SPILLED_RECORDS=13, GC_TIME_MILLIS=110, CPU_MILLISECONDS=0, PHYSICAL_MEMORY_BYTES=0, VIRTUAL_MEMORY_BYTES=0, COMMITTED_HEAP_BYTES=747634688, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=13, OUTPUT_BYTES=195, OUTPUT_BYTES_WITH_OVERHEAD=171, OUTPUT_BYTES_PHYSICAL=175, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=0, ADDITIONAL_SPILL_COUNT=0, cascading.flow.SliceCounters, Process_Begin_Time=1463523515672, Process_Duration=3, Process_End_Time=1463523515675, Read_Duration=1, Tuples_Read=13, Tuples_Written=13, Write_Duration=2, cascading.flow.StepCounters, Tuples_Read=13
2016-05-17 22:18:35,686 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:handle(692)) - attempt_1463523515295_0001_1_00_000000_0 TaskAttempt Transitioned from RUNNING to SUCCEEDED due to event TA_DONE
2016-05-17 22:18:35,687 INFO impl.TaskImpl (TaskImpl.java:transition(1052)) - Task succeeded with attempt attempt_1463523515295_0001_1_00_000000_0
2016-05-17 22:18:35,687 INFO rm.TaskSchedulerEventHandler (TaskSchedulerEventHandler.java:handleEvent(149)) - Processing the event EventType: S_TA_ENDED
2016-05-17 22:18:35,687 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event TASK_FINISHED to history file
2016-05-17 22:18:35,688 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1463523515295_0001_1][Event:TASK_FINISHED]: vertexName=BC1BA212A5D84497A413594BBD27B056, taskId=task_1463523515295_0001_1_00_000000, startTime=1463523515516, finishTime=1463523515687, timeTaken=171, status=SUCCEEDED, successfulAttemptID=attempt_1463523515295_0001_1_00_000000_0, diagnostics=, counters=Counters: 28, org.apache.tez.common.counters.DAGCounter, RACK_LOCAL_TASKS=1, File System Counters, FILE_BYTES_READ=110, FILE_BYTES_WRITTEN=727, FILE_READ_OPS=0, FILE_LARGE_READ_OPS=0, FILE_WRITE_OPS=0, org.apache.tez.common.counters.TaskCounter, SPILLED_RECORDS=13, GC_TIME_MILLIS=110, CPU_MILLISECONDS=0, PHYSICAL_MEMORY_BYTES=0, VIRTUAL_MEMORY_BYTES=0, COMMITTED_HEAP_BYTES=747634688, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=13, OUTPUT_BYTES=195, OUTPUT_BYTES_WITH_OVERHEAD=171, OUTPUT_BYTES_PHYSICAL=175, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=0, ADDITIONAL_SPILL_COUNT=0, cascading.flow.SliceCounters, Process_Begin_Time=1463523515672, Process_Duration=3, Process_End_Time=1463523515675, Read_Duration=1, Tuples_Read=13, Tuples_Written=13, Write_Duration=2, cascading.flow.StepCounters, Tuples_Read=13
2016-05-17 22:18:35,688 INFO impl.TaskImpl (TaskImpl.java:handle(862)) - task_1463523515295_0001_1_00_000000 Task Transitioned from RUNNING to SUCCEEDED due to event T_ATTEMPT_SUCCEEDED
2016-05-17 22:18:35,688 INFO impl.VertexImpl (VertexImpl.java:transition(3536)) - Num completed Tasks for vertex_1463523515295_0001_1_00 [BC1BA212A5D84497A413594BBD27B056] : 1
2016-05-17 22:18:35,688 INFO impl.VertexImpl (VertexImpl.java:checkVertexForCompletion(1674)) - Vertex succeeded: vertex_1463523515295_0001_1_00 [BC1BA212A5D84497A413594BBD27B056]
2016-05-17 22:18:35,689 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event VERTEX_FINISHED to history file
2016-05-17 22:18:35,689 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1463523515295_0001_1][Event:VERTEX_FINISHED]: vertexName=BC1BA212A5D84497A413594BBD27B056, vertexId=vertex_1463523515295_0001_1_00, initRequestedTime=1463523515473, initedTime=1463523515499, startRequestedTime=1463523515500, startedTime=1463523515500, finishTime=1463523515688, timeTaken=188, status=SUCCEEDED, diagnostics=, counters=Counters: 28, org.apache.tez.common.counters.DAGCounter, RACK_LOCAL_TASKS=1, File System Counters, FILE_BYTES_READ=110, FILE_BYTES_WRITTEN=727, FILE_READ_OPS=0, FILE_LARGE_READ_OPS=0, FILE_WRITE_OPS=0, org.apache.tez.common.counters.TaskCounter, SPILLED_RECORDS=13, GC_TIME_MILLIS=110, CPU_MILLISECONDS=0, PHYSICAL_MEMORY_BYTES=0, VIRTUAL_MEMORY_BYTES=0, COMMITTED_HEAP_BYTES=747634688, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=13, OUTPUT_BYTES=195, OUTPUT_BYTES_WITH_OVERHEAD=171, OUTPUT_BYTES_PHYSICAL=175, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=0, ADDITIONAL_SPILL_COUNT=0, cascading.flow.SliceCounters, Process_Begin_Time=1463523515672, Process_Duration=3, Process_End_Time=1463523515675, Read_Duration=1, Tuples_Read=13, Tuples_Written=13, Write_Duration=2, cascading.flow.StepCounters, Tuples_Read=13, vertexStats=firstTaskStartTime=1463523515516, firstTasksToStart=[ task_1463523515295_0001_1_00_000000 ], lastTaskFinishTime=1463523515686, lastTasksToFinish=[ task_1463523515295_0001_1_00_000000 ], minTaskDuration=170, maxTaskDuration=170, avgTaskDuration=170.0, numSuccessfulTasks=1, shortestDurationTasks=[ task_1463523515295_0001_1_00_000000 ], longestDurationTasks=[ task_1463523515295_0001_1_00_000000 ], vertexTaskStats={numFailedTasks=0, numSucceededTasks=1, numKilledTaskAttempts=0, numKilledTasks=0, numFailedTaskAttempts=0, numCompletedTasks=1}
2016-05-17 22:18:35,689 INFO impl.VertexImpl (VertexImpl.java:handle(1564)) - vertex_1463523515295_0001_1_00 [BC1BA212A5D84497A413594BBD27B056] transitioned from RUNNING to SUCCEEDED due to event V_TASK_COMPLETED
2016-05-17 22:18:35,690 INFO container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1463523515295_0001_00_000001 transitioned from RUNNING to IDLE via event C_TA_SUCCEEDED
2016-05-17 22:18:35,690 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:cleanup(712)) - Final Counters : Counters: 28 [[org.apache.tez.common.counters.DAGCounter RACK_LOCAL_TASKS=1][File System Counters FILE_BYTES_READ=110, FILE_BYTES_WRITTEN=727, FILE_READ_OPS=0, FILE_LARGE_READ_OPS=0, FILE_WRITE_OPS=0][org.apache.tez.common.counters.TaskCounter SPILLED_RECORDS=13, GC_TIME_MILLIS=110, CPU_MILLISECONDS=0, PHYSICAL_MEMORY_BYTES=0, VIRTUAL_MEMORY_BYTES=0, COMMITTED_HEAP_BYTES=747634688, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=13, OUTPUT_BYTES=195, OUTPUT_BYTES_WITH_OVERHEAD=171, OUTPUT_BYTES_PHYSICAL=175, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=0, ADDITIONAL_SPILL_COUNT=0][cascading.flow.SliceCounters Process_Begin_Time=1463523515672, Process_Duration=3, Process_End_Time=1463523515675, Read_Duration=1, Tuples_Read=13, Tuples_Written=13, Write_Duration=2][cascading.flow.StepCounters Tuples_Read=13]]
2016-05-17 22:18:35,690 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1463523515295_0001_1][Event:CONTAINER_STOPPED]: containerId=container_1463523515295_0001_00_000001, stoppedTime=1463523515690, exitStatus=0
2016-05-17 22:18:35,690 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(164)) - Redirecting log file based on addend:
2016-05-17 22:18:35,691 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(179)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-17 22:18:35,690 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event CONTAINER_STOPPED to history file
2016-05-17 22:18:35,691 INFO container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1463523515295_0001_00_000001 transitioned from IDLE to STOP_REQUESTED via event C_STOP_REQUEST
2016-05-17 22:18:35,691 INFO task.ContainerReporter (ContainerReporter.java:call(55)) - Attempting to fetch new task for container container_1463523515295_0001_00_000001
2016-05-17 22:18:35,691 INFO impl.VertexImpl (VertexImpl.java:transition(3477)) - Source task attempt completed for vertex: vertex_1463523515295_0001_1_01 [E045B238A1D94146A028712596348FCB] attempt: attempt_1463523515295_0001_1_00_000000_0 with state: SUCCEEDED vertexState: RUNNING
2016-05-17 22:18:35,692 INFO app.TaskAttemptListenerImpTezDag (TaskAttemptListenerImpTezDag.java:getTask(213)) - Container with id: container_1463523515295_0001_00_000001 is valid, but no longer registered, and will be killed
2016-05-17 22:18:35,692 INFO task.ContainerReporter (ContainerReporter.java:call(65)) - Got TaskUpdate for containerId= container_1463523515295_0001_00_000001: 0 ms after starting to poll. TaskInfo: shouldDie: true
2016-05-17 22:18:35,692 INFO task.TezChild (TezChild.java:run(207)) - ContainerTask returned shouldDie=true for container container_1463523515295_0001_00_000001, Exiting
2016-05-17 22:18:35,692 INFO task.TezChild (TezChild.java:shutdown(343)) - Shutdown invoked for container container_1463523515295_0001_00_000001
2016-05-17 22:18:35,692 INFO vertexmanager.ShuffleVertexManager (ShuffleVertexManager.java:schedulePendingTasks(608)) - All source tasks assigned. Ramping up 1 remaining tasks for vertex: E045B238A1D94146A028712596348FCB
2016-05-17 22:18:35,692 INFO task.TezChild (TezChild.java:shutdown(345)) - Shutting down container container_1463523515295_0001_00_000001
2016-05-17 22:18:35,692 INFO impl.VertexImpl (VertexImpl.java:scheduleTasks(1250)) - Routing pending task events for vertex: vertex_1463523515295_0001_1_01 [E045B238A1D94146A028712596348FCB]
2016-05-17 22:18:35,694 INFO impl.DAGImpl (DAGImpl.java:transition(1747)) - Vertex vertex_1463523515295_0001_1_00 [BC1BA212A5D84497A413594BBD27B056] completed., numCompletedVertices=1, numSuccessfulVertices=1, numFailedVertices=0, numKilledVertices=0, numVertices=2
2016-05-17 22:18:35,695 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event TASK_STARTED to history file
2016-05-17 22:18:35,695 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1463523515295_0001_1][Event:TASK_STARTED]: vertexName=E045B238A1D94146A028712596348FCB, taskId=task_1463523515295_0001_1_01_000000, scheduledTime=1463523515695, launchTime=1463523515695
2016-05-17 22:18:35,695 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:stop(244)) - Ignoring stop request for containerId container_1463523515295_0001_00_000001. Relying on regular task shutdown for it to end
2016-05-17 22:18:35,695 INFO impl.TaskImpl (TaskImpl.java:handle(862)) - task_1463523515295_0001_1_01_000000 Task Transitioned from NEW to SCHEDULED due to event T_SCHEDULE
2016-05-17 22:18:35,695 INFO impl.DAGSchedulerNaturalOrderControlled (DAGSchedulerNaturalOrderControlled.java:scheduleTask(111)) - Scheduled vertex: vertex_1463523515295_0001_1_01 [E045B238A1D94146A028712596348FCB]
2016-05-17 22:18:35,695 INFO container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1463523515295_0001_00_000001 transitioned from STOP_REQUESTED to STOPPING via event C_NM_STOP_SENT
2016-05-17 22:18:35,696 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:transition(1014)) - remoteTaskSpec:DAGName : [B5FF3D6B3FEC4DE2963B32969B26C7D0/A34225503819433D86E0612345749768] test/(1/1) ...m/testFirstNBuffer/firstn, VertexName: E045B238A1D94146A028712596348FCB, VertexParallelism: 1, TaskAttemptID:attempt_1463523515295_0001_1_01_000000_0, processorName=cascading.flow.tez.FlowProcessor, inputSpecListSize=1, outputSpecListSize=1, inputSpecList=[{{ sourceVertexName=BC1BA212A5D84497A413594BBD27B056, physicalEdgeCount=1, inputClassName=org.apache.tez.runtime.library.input.OrderedGroupedKVInput }}, ], outputSpecList=[{{ destinationVertexName=26F4A514C34C40A88954A24ACAF1AF82, physicalEdgeCount=0, outputClassName=org.apache.tez.mapreduce.output.MROutput }}, ]
2016-05-17 22:18:35,696 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:handle(692)) - attempt_1463523515295_0001_1_01_000000_0 TaskAttempt Transitioned from NEW to START_WAIT due to event TA_SCHEDULE
2016-05-17 22:18:35,696 INFO rm.TaskSchedulerEventHandler (TaskSchedulerEventHandler.java:handleEvent(149)) - Processing the event EventType: S_TA_LAUNCH_REQUEST
2016-05-17 22:18:35,697 INFO container.AMContainerImpl (AMContainerImpl.java:transition(521)) - Sending Launch Request for Container with id: container_1463523515295_0001_00_000002
2016-05-17 22:18:35,697 INFO container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1463523515295_0001_00_000002 transitioned from ALLOCATED to LAUNCHING via event C_LAUNCH_REQUEST
2016-05-17 22:18:35,698 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(270)) - ContainerExecutionResult for: container_1463523515295_0001_00_000001 = ContainerExecutionResult{exitStatus=SUCCESS, throwable=null, errorMessage='Asked to die by the AM'}
2016-05-17 22:18:35,698 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(274)) - Container: container_1463523515295_0001_00_000001 completed successfully
2016-05-17 22:18:35,699 INFO task.TezChild (TezChild.java:newTezChild(432)) - PID, containerIdentifier: null, container_1463523515295_0001_00_000002
2016-05-17 22:18:35,699 INFO container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1463523515295_0001_00_000001 transitioned from STOPPING to COMPLETED via event C_COMPLETED
2016-05-17 22:18:35,699 WARN impl.MetricsConfig (MetricsConfig.java:loadFirst(124)) - Cannot locate configuration: tried hadoop-metrics2-teztask.properties,hadoop-metrics2.properties
2016-05-17 22:18:35,702 INFO container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1463523515295_0001_00_000002 transitioned from LAUNCHING to IDLE via event C_LAUNCHED
2016-05-17 22:18:35,702 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event CONTAINER_LAUNCHED to history file
2016-05-17 22:18:35,702 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1463523515295_0001_1][Event:CONTAINER_LAUNCHED]: containerId=container_1463523515295_0001_00_000002, launchTime=1463523515702
2016-05-17 22:18:35,703 INFO task.ContainerReporter (ContainerReporter.java:call(55)) - Attempting to fetch new task for container container_1463523515295_0001_00_000002
2016-05-17 22:18:35,704 INFO container.AMContainerImpl (AMContainerImpl.java:transition(827)) - Assigned taskAttempt + [attempt_1463523515295_0001_1_01_000000_0] to container: [container_1463523515295_0001_00_000002]
2016-05-17 22:18:35,704 INFO container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1463523515295_0001_00_000002 transitioned from IDLE to RUNNING via event C_PULL_TA
2016-05-17 22:18:35,704 INFO app.TaskAttemptListenerImpTezDag (TaskAttemptListenerImpTezDag.java:getTask(240)) - Container with id: container_1463523515295_0001_00_000002 given task: attempt_1463523515295_0001_1_01_000000_0
2016-05-17 22:18:35,704 INFO task.ContainerReporter (ContainerReporter.java:call(65)) - Got TaskUpdate for containerId= container_1463523515295_0001_00_000002: 0 ms after starting to poll. TaskInfo: shouldDie: false, currentTaskAttemptId: attempt_1463523515295_0001_1_01_000000_0
2016-05-17 22:18:35,705 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:transition(1146)) - TaskAttempt: [attempt_1463523515295_0001_1_01_000000_0] started. Is using containerId: [container_1463523515295_0001_00_000002] on NM: [127.0.0.1:0]
2016-05-17 22:18:35,705 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event TASK_ATTEMPT_STARTED to history file
2016-05-17 22:18:35,705 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1463523515295_0001_1][Event:TASK_ATTEMPT_STARTED]: vertexName=E045B238A1D94146A028712596348FCB, taskAttemptId=attempt_1463523515295_0001_1_01_000000_0, startTime=1463523515704, containerId=container_1463523515295_0001_00_000002, nodeId=127.0.0.1:0, inProgressLogs=127.0.0.1:0/node/containerlogs/container_1463523515295_0001_00_000002/teamcity, completedLogs=
2016-05-17 22:18:35,705 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:handle(692)) - attempt_1463523515295_0001_1_01_000000_0 TaskAttempt Transitioned from START_WAIT to RUNNING due to event TA_STARTED_REMOTELY
2016-05-17 22:18:35,705 INFO impl.TaskImpl (TaskImpl.java:handle(862)) - task_1463523515295_0001_1_01_000000 Task Transitioned from SCHEDULED to RUNNING due to event T_ATTEMPT_LAUNCHED
2016-05-17 22:18:35 Starting to run new task attempt: attempt_1463523515295_0001_1_01_000000_0
2016-05-17 22:18:35,706 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(164)) - Redirecting log file based on addend: attempt_1463523515295_0001_1_01_000000_0
2016-05-17 22:18:35,706 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(179)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-17 22:18:35,706 INFO task.TezChild (TezChild.java:handleNewTaskCredentials(273)) - Refreshing UGI since Credentials have changed
2016-05-17 22:18:35,707 INFO task.TezChild (TezChild.java:handleNewTaskCredentials(276)) - Credentials : #Tokens=1, #SecretKeys=0
2016-05-17 22:18:35,707 INFO task.TezChild (TezChild.java:handleNewTaskLocalResources(302)) - Localizing additional local resources for Task : {}
2016-05-17 22:18:35,707 INFO task.TezChild (TezChild.java:handleNewTaskLocalResources(312)) - Done localizing additional resources
2016-05-17 22:18:35,708 INFO metrics.TaskCounterUpdater (TaskCounterUpdater.java:initResourceCalculatorPlugin(148)) - Using ResourceCalculatorProcessTree : [ ]
2016-05-17 22:18:35,708 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:<init>(149)) - Initializing LogicalIOProcessorRuntimeTask with TaskSpec: DAGName : [B5FF3D6B3FEC4DE2963B32969B26C7D0/A34225503819433D86E0612345749768] test/(1/1) ...m/testFirstNBuffer/firstn, VertexName: E045B238A1D94146A028712596348FCB, VertexParallelism: 1, TaskAttemptID:attempt_1463523515295_0001_1_01_000000_0, processorName=cascading.flow.tez.FlowProcessor, inputSpecListSize=1, outputSpecListSize=1, inputSpecList=[{{ sourceVertexName=BC1BA212A5D84497A413594BBD27B056, physicalEdgeCount=1, inputClassName=org.apache.tez.runtime.library.input.OrderedGroupedKVInput }}, ], outputSpecList=[{{ destinationVertexName=26F4A514C34C40A88954A24ACAF1AF82, physicalEdgeCount=0, outputClassName=org.apache.tez.mapreduce.output.MROutput }}, ]
2016-05-17 22:18:35,708 INFO resources.MemoryDistributor (MemoryDistributor.java:<init>(85)) - InitialMemoryDistributor (isEnabled=true) invoked with: numInputs=1, numOutputs=1, JVM.maxFree=747634688
2016-05-17 22:18:35,709 INFO task.TezTaskRunner (TezTaskRunner.java:run(172)) - Initializing task, taskAttemptId=attempt_1463523515295_0001_1_01_000000_0
2016-05-17 22:18:35,710 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(191)) - Initializing LogicalProcessorIORuntimeTask
2016-05-17 22:18:35,710 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(195)) - Creating processor, processorClassName=cascading.flow.tez.FlowProcessor
2016-05-17 22:18:35,710 INFO impl.TaskImpl (TaskImpl.java:getTaskAttemptTezEvents(481)) - TaskAttempt:attempt_1463523515295_0001_1_01_000000_0 sent events: (0-1)
2016-05-17 22:18:35,711 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initializeLogicalIOProcessor(488)) - Initializing processor, processorClassName=cascading.flow.tez.FlowProcessor
2016-05-17 22:18:35,711 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(389)) - Initializing Input using InputSpec: { sourceVertexName=BC1BA212A5D84497A413594BBD27B056, physicalEdgeCount=1, inputClassName=org.apache.tez.runtime.library.input.OrderedGroupedKVInput }
2016-05-17 22:18:35,712 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:createInput(537)) - Creating Input
2016-05-17 22:18:35,712 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(397)) - Initializing Input with src edge: BC1BA212A5D84497A413594BBD27B056
2016-05-17 22:18:35,712 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(438)) - Initializing Output using OutputSpec: { destinationVertexName=26F4A514C34C40A88954A24ACAF1AF82, physicalEdgeCount=0, outputClassName=org.apache.tez.mapreduce.output.MROutput }
2016-05-17 22:18:35,713 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:createOutput(565)) - Creating Output
2016-05-17 22:18:35,713 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(446)) - Initializing Output with dest edge: 26F4A514C34C40A88954A24ACAF1AF82
2016-05-17 22:18:35,714 INFO output.MROutput (MROutput.java:initialize(347)) - Initializing Simple Output
2016-05-17 22:18:35,714 INFO tez.FlowProcessor (FlowProcessor.java:initialize(88)) - cascading version: 3.1.0
2016-05-17 22:18:35,714 INFO resources.MemoryDistributor$RequestorInfo (MemoryDistributor.java:<init>(233)) - Received request: 0, type: OUTPUT, componentVertexName: 26F4A514C34C40A88954A24ACAF1AF82
2016-05-17 22:18:35,715 INFO orderedgrouped.MergeManager (MergeManager.java:getInitialMemoryRequirement(310)) - Initial Shuffle Memory Required: 672871232, based on INPUT_BUFFER_factor: 0.9
2016-05-17 22:18:35,715 INFO orderedgrouped.MergeManager (MergeManager.java:getInitialMemoryRequirement(320)) - Initial Memory required for final merged output: 0, using factor: 0.0
2016-05-17 22:18:35,716 INFO resources.MemoryDistributor$RequestorInfo (MemoryDistributor.java:<init>(233)) - Received request: 672871232, type: INPUT, componentVertexName: BC1BA212A5D84497A413594BBD27B056
2016-05-17 22:18:35,718 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(403)) - Initialized Input with src edge: BC1BA212A5D84497A413594BBD27B056
2016-05-17 22:18:35,725 INFO tez.FlowProcessor (FlowProcessor.java:initialize(94)) - flow node id: E045B238A1D94146A028712596348FCB, ordinal: 1
2016-05-17 22:18:35,726 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: E045B238A1D94146A028712596348FCB, mem on start (mb), free: 579, total: 713, max: 713
2016-05-17 22:18:35,726 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initializeLogicalIOProcessor(491)) - Initialized processor, processorClassName=cascading.flow.tez.FlowProcessor
2016-05-17 22:18:35,727 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(220)) - Waiting for 2 initializers to finish
2016-05-17 22:18:35,727 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(220)) - Waiting for 1 initializers to finish
2016-05-17 22:18:35,730 INFO output.MROutput (MROutput.java:initialize(418)) - Initialized Simple Output, using_new_api: false
2016-05-17 22:18:35,730 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(452)) - Initialized Output with dest edge: 26F4A514C34C40A88954A24ACAF1AF82
2016-05-17 22:18:35,730 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(233)) - All initializers finished
2016-05-17 22:18:35,730 INFO resources.MemoryDistributor (MemoryDistributor.java:makeInitialAllocations(124)) - Using Allocator class: org.apache.tez.runtime.library.resources.WeightedScalingMemoryDistributor
2016-05-17 22:18:35,731 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:getRequestTypeForClass(197)) - Falling back to RequestType.OTHER for class: org.apache.tez.mapreduce.output.MROutput
2016-05-17 22:18:35,731 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:initialProcessMemoryRequestContext(171)) - ScaleFactor: 1, for type: OTHER
2016-05-17 22:18:35,731 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:initialProcessMemoryRequestContext(171)) - ScaleFactor: 12, for type: SORTED_MERGED_INPUT
2016-05-17 22:18:35,731 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:computeReservedFraction(262)) - InitialReservationFraction=0.3, AdditionalReservationFractionForIOs=0.03, finalReserveFractionUsed=0.32999999999999996
2016-05-17 22:18:35,731 INFO resources.WeightedScalingMemoryDistributor (WeightedScalingMemoryDistributor.java:assignMemory(126)) - Scaling Requests. NumRequests: 2, numScaledRequests: 13, TotalRequested: 672871232, TotalRequestedScaled: 6.211119064615386E8, TotalJVMHeap: 747634688, TotalAvailable: 500915240, TotalRequested/TotalJVMHeap:0.90
2016-05-17 22:18:35,731 INFO resources.MemoryDistributor (MemoryDistributor.java:makeInitialAllocations(138)) - Informing: OUTPUT, 26F4A514C34C40A88954A24ACAF1AF82, org.apache.tez.mapreduce.output.MROutput: requested=0, allocated=0
2016-05-17 22:18:35,731 INFO resources.MemoryDistributor (MemoryDistributor.java:makeInitialAllocations(138)) - Informing: INPUT, BC1BA212A5D84497A413594BBD27B056, org.apache.tez.runtime.library.input.OrderedGroupedKVInput: requested=672871232, allocated=500915240
2016-05-17 22:18:35,731 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(255)) - Starting Inputs/Outputs
2016-05-17 22:18:35,731 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(268)) - Input: BC1BA212A5D84497A413594BBD27B056 being auto started by the framework. Subsequent instances will not be auto-started
2016-05-17 22:18:35,731 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(419)) - Starting Input with src edge: BC1BA212A5D84497A413594BBD27B056
2016-05-17 22:18:35,732 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(289)) - Num IOs determined for AutoStart: 1
2016-05-17 22:18:35,732 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(291)) - Waiting for 1 IOs to start
2016-05-17 22:18:35,732 INFO common.TezRuntimeUtils (TezRuntimeUtils.java:instantiateCombiner(65)) - No combiner specified via tez.runtime.combiner.class. Combiner will not be used
2016-05-17 22:18:35,733 INFO orderedgrouped.Shuffle (Shuffle.java:<init>(182)) - Shuffle assigned with 1 inputs, codec: NoneifileReadAhead: true
2016-05-17 22:18:35,733 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:<init>(158)) - ShuffleScheduler running for sourceVertex: BC1BA212A5D84497A413594BBD27B056 with configuration: maxFetchFailuresBeforeReporting=5, reportReadErrorImmediately=true, maxFailedUniqueFetches=1, abortFailureLimit=30, maxMapRuntime=0, maxTaskOutputAtOnce=20
2016-05-17 22:18:35,733 INFO orderedgrouped.MergeManager (MergeManager.java:<init>(221)) - InitialRequest: ShuffleMem=672871232, postMergeMem=0, RuntimeTotalAvailable=500915240. Updated to: ShuffleMem=500915240, postMergeMem: 0
2016-05-17 22:18:35,733 INFO orderedgrouped.MergeManager (MergeManager.java:<init>(252)) - MergerManager: memoryLimit=500915240, maxSingleShuffleLimit=125228808, mergeThreshold=450823712, ioSortFactor=100, memToMemMergeOutputsThreshold=100
2016-05-17 22:18:35,733 INFO orderedgrouped.Shuffle (Shuffle.java:<init>(233)) - Num fetchers being started: 1
2016-05-17 22:18:35,734 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:call(421)) - Started Input with src edge: BC1BA212A5D84497A413594BBD27B056
2016-05-17 22:18:35,734 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:initialize(304)) - AutoStartComplete
2016-05-17 22:18:35,735 INFO task.TezTaskRunner (TezTaskRunner.java:run(175)) - Running task, taskAttemptId=attempt_1463523515295_0001_1_01_000000_0
2016-05-17 22:18:35,738 INFO graph.Hadoop2TezStreamGraph (Hadoop2TezStreamGraph.java:buildGraph(151)) - using streamed source: GroupBy(test)[by:[{1}:'num']]
2016-05-17 22:18:35,738 INFO tez.FlowProcessor (FlowProcessor.java:run(121)) - sourcing from: GroupBy(test)[by:[{1}:'num']] streamed: true, id: CE3CB083A5F44E699FADD0A5DDB4937D
2016-05-17 22:18:35,739 INFO tez.FlowProcessor (FlowProcessor.java:run(124)) - sinking to: Hfs["TextDelimited[['num', 'lower']]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.5.x-tez-0.6.x/build/test/output/hadoop2-tez/bufferpipesplatform/testFirstNBuffer/firstn"], id: 26F4A514C34C40A88954A24ACAF1AF82
2016-05-17 22:18:35,739 INFO element.TezSinkStage (TezSinkStage.java:prepare(57)) - calling MROutput#start() on: Hfs["TextDelimited[['num', 'lower']]"]["/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.5.x-tez-0.6.x/build/test/output/hadoop2-tez/bufferpipesplatform/testFirstNBuffer/firstn"]
2016-05-17 22:18:35,739 INFO element.TezGroupGate (TezGroupGate.java:prepare(90)) - calling OrderedGroupedKVInput#start() on: GroupBy(test)[by:[{1}:'num']] CE3CB083A5F44E699FADD0A5DDB4937D, for 1 inputs
2016-05-17 22:18:35,740 INFO orderedgrouped.ShuffleInputEventHandlerOrderedGrouped (ShuffleInputEventHandlerOrderedGrouped.java:processDataMovementEvent(84)) - DME srcIdx: 0, targetIdx: 0, attemptNum: 0, payload: [host: ip-10-123-148-208, port: 0, pathComponent: attempt_1463523515295_0001_1_00_000000_0_10049, runDuration: 160970, ]
2016-05-17 22:18:35,742 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:logIndividualFetchComplete(250)) - Completed fetch for attempt: InputAttemptIdentifier [inputIdentifier=InputIdentifier [inputIndex=0], attemptNumber=0, pathComponent=attempt_1463523515295_0001_1_00_000000_0_10049] to DISK_DIRECT, CompressedSize=175, DecompressedSize=171,EndTime=1463523515742, TimeTaken=1, Rate=0.17 MB/s
2016-05-17 22:18:35,742 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:copySucceeded(213)) - All inputs fetched for input vertex : BC1BA212A5D84497A413594BBD27B056
2016-05-17 22:18:35,743 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:logProgress(263)) - copy(1 of 1. Transfer rate (CumulativeDataFetched/TimeSinceInputStarted)) 0.00 MB/s)
2016-05-17 22:18:35,743 INFO orderedgrouped.ShuffleScheduler (ShuffleScheduler.java:freeHost(529)) - ip-10-123-148-208:0 freed by fetcher [BC1BA212A5D84497A413594BBD27B056] #9 in 2ms
2016-05-17 22:18:35,743 INFO orderedgrouped.Shuffle (Shuffle.java:cleanupFetchers(394)) - Shutdown..fetcher [BC1BA212A5D84497A413594BBD27B056] #9, status:false, isInterrupted:false
2016-05-17 22:18:35,744 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(854)) - finalMerge called with 0 in-memory map-outputs and 1 on-disk map-outputs
2016-05-17 22:18:35,745 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(951)) - Merging 1 files, 175 bytes from disk
2016-05-17 22:18:35,745 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(966)) - Merging 0 segments, 0 bytes from memory into reduce
2016-05-17 22:18:35,745 INFO impl.TezMerger (TezMerger.java:merge(560)) - Merging 1 sorted segments
2016-05-17 22:18:35,745 INFO impl.TezMerger (TezMerger.java:merge(665)) - Down to the last merge-pass, with 1 segments left of total size: 158 bytes
2016-05-17 22:18:35,745 INFO orderedgrouped.Shuffle (Shuffle.java:call(381)) - merge complete for input vertex : BC1BA212A5D84497A413594BBD27B056
2016-05-17 22:18:35,746 INFO orderedgrouped.Shuffle (Shuffle.java:onSuccess(488)) - Shuffle Runner thread complete
2016-05-17 22:18:35,746 INFO tez.FlowProcessor (FlowProcessor.java:waitForInputsReady(202)) - flow node id: E045B238A1D94146A028712596348FCB, all 1 inputs ready in: 00:00:00.007
2016-05-17 22:18:35,748 INFO common.ValuesIterator (ValuesIterator.java:<init>(83)) - keyDeserializer=cascading.tuple.hadoop.io.KeyTupleDeserializer@1fa819a4; valueDeserializer=cascading.tuple.hadoop.io.ValueTupleDeserializer@552b6e38
2016-05-17 22:18:35,751 INFO output.MROutput (MROutput.java:flush(534)) - Flushing Simple Output
2016-05-17 22:18:35,751 INFO output.MROutput (MROutput.java:flush(544)) - Flushed Simple Output
2016-05-17 22:18:35,752 INFO app.TaskAttemptListenerImpTezDag (TaskAttemptListenerImpTezDag.java:canCommit(353)) - Commit go/no-go request from attempt_1463523515295_0001_1_01_000000_0
2016-05-17 22:18:35,752 INFO impl.TaskImpl (TaskImpl.java:canCommit(750)) - attempt_1463523515295_0001_1_01_000000_0 given a go for committing the task output.
2016-05-17 22:18:35,753 INFO task.TezTaskRunner (TezTaskRunner.java:run(177)) - Closing task, taskAttemptId=attempt_1463523515295_0001_1_01_000000_0
2016-05-17 22:18:35,753 INFO tez.FlowProcessor (LogUtil.java:logMemory(59)) - flow node id: E045B238A1D94146A028712596348FCB, mem on close (mb), free: 576, total: 713, max: 713
2016-05-17 22:18:35,754 INFO orderedgrouped.Shuffle (Shuffle.java:shutdown(320)) - Shutting down Shuffle for source: BC1BA212A5D84497A413594BBD27B056
2016-05-17 22:18:35,754 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(854)) - finalMerge called with 0 in-memory map-outputs and 0 on-disk map-outputs
2016-05-17 22:18:35,755 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(951)) - Merging 0 files, 0 bytes from disk
2016-05-17 22:18:35,755 INFO orderedgrouped.MergeManager (MergeManager.java:finalMerge(966)) - Merging 0 segments, 0 bytes from memory into reduce
2016-05-17 22:18:35,755 INFO impl.TezMerger (TezMerger.java:merge(560)) - Merging 0 sorted segments
2016-05-17 22:18:35,756 INFO impl.TezMerger (TezMerger.java:merge(562)) - Nothing to merge. Returning an empty iterator
2016-05-17 22:18:35 Completed running task attempt: attempt_1463523515295_0001_1_01_000000_0
2016-05-17 22:18:35,757 INFO task.TezTaskRunner (TezTaskRunner.java:run(181)) - Task completed, taskAttemptId=attempt_1463523515295_0001_1_01_000000_0, fatalErrorOccurred=false
2016-05-17 22:18:35,758 INFO runtime.LogicalIOProcessorRuntimeTask (LogicalIOProcessorRuntimeTask.java:cleanup(712)) - Final Counters : Counters: 46 [[File System Counters FILE_BYTES_READ=207, FILE_BYTES_WRITTEN=32, FILE_READ_OPS=0, FILE_LARGE_READ_OPS=0, FILE_WRITE_OPS=0][org.apache.tez.common.counters.TaskCounter REDUCE_INPUT_GROUPS=4, REDUCE_INPUT_RECORDS=5, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=13, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, GC_TIME_MILLIS=0, CPU_MILLISECONDS=0, PHYSICAL_MEMORY_BYTES=0, VIRTUAL_MEMORY_BYTES=0, COMMITTED_HEAP_BYTES=747634688, OUTPUT_RECORDS=5, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=175, SHUFFLE_BYTES=175, SHUFFLE_BYTES_DECOMPRESSED=171, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=175, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=10, MERGE_PHASE_TIME=12, FIRST_EVENT_RECEIVED=8, LAST_EVENT_RECEIVED=8][Shuffle Errors BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0][cascading.flow.SliceCounters Process_Begin_Time=1463523515746, Process_Duration=7, Process_End_Time=1463523515753, Read_Duration=1, Tuples_Read=5, Tuples_Written=5, Write_Duration=0][cascading.flow.StepCounters Tuples_Written=5]]
2016-05-17 22:18:35,759 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(164)) - Redirecting log file based on addend:
2016-05-17 22:18:35,759 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(179)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-17 22:18:35,758 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1463523515295_0001_1][Event:TASK_ATTEMPT_FINISHED]: vertexName=E045B238A1D94146A028712596348FCB, taskAttemptId=attempt_1463523515295_0001_1_01_000000_0, startTime=1463523515704, finishTime=1463523515758, timeTaken=54, status=SUCCEEDED, errorEnum=, diagnostics=, counters=Counters: 46, File System Counters, FILE_BYTES_READ=207, FILE_BYTES_WRITTEN=32, FILE_READ_OPS=0, FILE_LARGE_READ_OPS=0, FILE_WRITE_OPS=0, org.apache.tez.common.counters.TaskCounter, REDUCE_INPUT_GROUPS=4, REDUCE_INPUT_RECORDS=5, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=13, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, GC_TIME_MILLIS=0, CPU_MILLISECONDS=0, PHYSICAL_MEMORY_BYTES=0, VIRTUAL_MEMORY_BYTES=0, COMMITTED_HEAP_BYTES=747634688, OUTPUT_RECORDS=5, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=175, SHUFFLE_BYTES=175, SHUFFLE_BYTES_DECOMPRESSED=171, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=175, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=10, MERGE_PHASE_TIME=12, FIRST_EVENT_RECEIVED=8, LAST_EVENT_RECEIVED=8, Shuffle Errors, BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0, cascading.flow.SliceCounters, Process_Begin_Time=1463523515746, Process_Duration=7, Process_End_Time=1463523515753, Read_Duration=1, Tuples_Read=5, Tuples_Written=5, Write_Duration=0, cascading.flow.StepCounters, Tuples_Written=5
2016-05-17 22:18:35,758 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event TASK_ATTEMPT_FINISHED to history file
2016-05-17 22:18:35,760 INFO impl.TaskAttemptImpl (TaskAttemptImpl.java:handle(692)) - attempt_1463523515295_0001_1_01_000000_0 TaskAttempt Transitioned from RUNNING to SUCCEEDED due to event TA_DONE
2016-05-17 22:18:35,760 INFO impl.TaskImpl (TaskImpl.java:transition(1052)) - Task succeeded with attempt attempt_1463523515295_0001_1_01_000000_0
2016-05-17 22:18:35,760 INFO rm.TaskSchedulerEventHandler (TaskSchedulerEventHandler.java:handleEvent(149)) - Processing the event EventType: S_TA_ENDED
2016-05-17 22:18:35,759 INFO task.ContainerReporter (ContainerReporter.java:call(55)) - Attempting to fetch new task for container container_1463523515295_0001_00_000002
2016-05-17 22:18:35,760 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event TASK_FINISHED to history file
2016-05-17 22:18:35,761 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1463523515295_0001_1][Event:TASK_FINISHED]: vertexName=E045B238A1D94146A028712596348FCB, taskId=task_1463523515295_0001_1_01_000000, startTime=1463523515704, finishTime=1463523515760, timeTaken=56, status=SUCCEEDED, successfulAttemptID=attempt_1463523515295_0001_1_01_000000_0, diagnostics=, counters=Counters: 46, File System Counters, FILE_BYTES_READ=207, FILE_BYTES_WRITTEN=32, FILE_READ_OPS=0, FILE_LARGE_READ_OPS=0, FILE_WRITE_OPS=0, org.apache.tez.common.counters.TaskCounter, REDUCE_INPUT_GROUPS=4, REDUCE_INPUT_RECORDS=5, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=13, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, GC_TIME_MILLIS=0, CPU_MILLISECONDS=0, PHYSICAL_MEMORY_BYTES=0, VIRTUAL_MEMORY_BYTES=0, COMMITTED_HEAP_BYTES=747634688, OUTPUT_RECORDS=5, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=175, SHUFFLE_BYTES=175, SHUFFLE_BYTES_DECOMPRESSED=171, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=175, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=10, MERGE_PHASE_TIME=12, FIRST_EVENT_RECEIVED=8, LAST_EVENT_RECEIVED=8, Shuffle Errors, BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0, cascading.flow.SliceCounters, Process_Begin_Time=1463523515746, Process_Duration=7, Process_End_Time=1463523515753, Read_Duration=1, Tuples_Read=5, Tuples_Written=5, Write_Duration=0, cascading.flow.StepCounters, Tuples_Written=5
2016-05-17 22:18:35,761 INFO impl.TaskImpl (TaskImpl.java:handle(862)) - task_1463523515295_0001_1_01_000000 Task Transitioned from RUNNING to SUCCEEDED due to event T_ATTEMPT_SUCCEEDED
2016-05-17 22:18:35,761 INFO impl.VertexImpl (VertexImpl.java:transition(3536)) - Num completed Tasks for vertex_1463523515295_0001_1_01 [E045B238A1D94146A028712596348FCB] : 1
2016-05-17 22:18:35,761 INFO impl.VertexImpl (VertexImpl.java:checkVertexForCompletion(1674)) - Vertex succeeded: vertex_1463523515295_0001_1_01 [E045B238A1D94146A028712596348FCB]
2016-05-17 22:18:35,763 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1463523515295_0001_1][Event:VERTEX_FINISHED]: vertexName=E045B238A1D94146A028712596348FCB, vertexId=vertex_1463523515295_0001_1_01, initRequestedTime=1463523515475, initedTime=1463523515492, startRequestedTime=1463523515505, startedTime=1463523515505, finishTime=1463523515761, timeTaken=256, status=SUCCEEDED, diagnostics=, counters=Counters: 46, File System Counters, FILE_BYTES_READ=207, FILE_BYTES_WRITTEN=32, FILE_READ_OPS=0, FILE_LARGE_READ_OPS=0, FILE_WRITE_OPS=0, org.apache.tez.common.counters.TaskCounter, REDUCE_INPUT_GROUPS=4, REDUCE_INPUT_RECORDS=5, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=13, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, GC_TIME_MILLIS=0, CPU_MILLISECONDS=0, PHYSICAL_MEMORY_BYTES=0, VIRTUAL_MEMORY_BYTES=0, COMMITTED_HEAP_BYTES=747634688, OUTPUT_RECORDS=5, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=175, SHUFFLE_BYTES=175, SHUFFLE_BYTES_DECOMPRESSED=171, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=175, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=10, MERGE_PHASE_TIME=12, FIRST_EVENT_RECEIVED=8, LAST_EVENT_RECEIVED=8, Shuffle Errors, BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0, cascading.flow.SliceCounters, Process_Begin_Time=1463523515746, Process_Duration=7, Process_End_Time=1463523515753, Read_Duration=1, Tuples_Read=5, Tuples_Written=5, Write_Duration=0, cascading.flow.StepCounters, Tuples_Written=5, vertexStats=firstTaskStartTime=1463523515704, firstTasksToStart=[ task_1463523515295_0001_1_01_000000 ], lastTaskFinishTime=1463523515758, lastTasksToFinish=[ task_1463523515295_0001_1_01_000000 ], minTaskDuration=54, maxTaskDuration=54, avgTaskDuration=54.0, numSuccessfulTasks=1, shortestDurationTasks=[ task_1463523515295_0001_1_01_000000 ], longestDurationTasks=[ task_1463523515295_0001_1_01_000000 ], vertexTaskStats={numFailedTasks=0, numSucceededTasks=1, numKilledTaskAttempts=0, numKilledTasks=0, numFailedTaskAttempts=0, numCompletedTasks=1}
2016-05-17 22:18:35,763 INFO impl.VertexImpl (VertexImpl.java:handle(1564)) - vertex_1463523515295_0001_1_01 [E045B238A1D94146A028712596348FCB] transitioned from RUNNING to SUCCEEDED due to event V_TASK_COMPLETED
2016-05-17 22:18:35,763 INFO container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1463523515295_0001_00_000002 transitioned from RUNNING to IDLE via event C_TA_SUCCEEDED
2016-05-17 22:18:35,763 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event VERTEX_FINISHED to history file
2016-05-17 22:18:35,765 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1463523515295_0001_1][Event:CONTAINER_STOPPED]: containerId=container_1463523515295_0001_00_000002, stoppedTime=1463523515765, exitStatus=0
2016-05-17 22:18:35,766 INFO container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1463523515295_0001_00_000002 transitioned from IDLE to STOP_REQUESTED via event C_STOP_REQUEST
2016-05-17 22:18:35,766 INFO impl.DAGImpl (DAGImpl.java:transition(1747)) - Vertex vertex_1463523515295_0001_1_01 [E045B238A1D94146A028712596348FCB] completed., numCompletedVertices=2, numSuccessfulVertices=2, numFailedVertices=0, numKilledVertices=0, numVertices=2
2016-05-17 22:18:35,767 INFO impl.DAGImpl (DAGImpl.java:commitOrAbortOutputs(831)) - Calling DAG commit/abort for dag: dag_1463523515295_0001_1
2016-05-17 22:18:35,765 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event CONTAINER_STOPPED to history file
2016-05-17 22:18:35,767 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1463523515295_0001_1][Event:DAG_COMMIT_STARTED]: dagID=dag_1463523515295_0001_1
2016-05-17 22:18:35,768 INFO impl.DAGImpl (DAGImpl.java:commitOrAbortOutputs(871)) - No output committers for vertex: vertex_1463523515295_0001_1_00 [BC1BA212A5D84497A413594BBD27B056]
2016-05-17 22:18:35,768 INFO impl.DAGImpl (DAGImpl.java:commitOrAbortOutputs(892)) - Committing output: 26F4A514C34C40A88954A24ACAF1AF82 for vertex: vertex_1463523515295_0001_1_01 [E045B238A1D94146A028712596348FCB]
2016-05-17 22:18:35,771 INFO app.TaskAttemptListenerImpTezDag (TaskAttemptListenerImpTezDag.java:getTask(213)) - Container with id: container_1463523515295_0001_00_000002 is valid, but no longer registered, and will be killed
2016-05-17 22:18:35,771 INFO task.ContainerReporter (ContainerReporter.java:call(65)) - Got TaskUpdate for containerId= container_1463523515295_0001_00_000002: 10 ms after starting to poll. TaskInfo: shouldDie: true
2016-05-17 22:18:35,771 INFO task.TezChild (TezChild.java:run(207)) - ContainerTask returned shouldDie=true for container container_1463523515295_0001_00_000002, Exiting
2016-05-17 22:18:35,771 INFO task.TezChild (TezChild.java:shutdown(343)) - Shutdown invoked for container container_1463523515295_0001_00_000002
2016-05-17 22:18:35,771 INFO task.TezChild (TezChild.java:shutdown(345)) - Shutting down container container_1463523515295_0001_00_000002
2016-05-17 22:18:35,772 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(270)) - ContainerExecutionResult for: container_1463523515295_0001_00_000002 = ContainerExecutionResult{exitStatus=SUCCESS, throwable=null, errorMessage='Asked to die by the AM'}
2016-05-17 22:18:35,772 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:onSuccess(274)) - Container: container_1463523515295_0001_00_000002 completed successfully
2016-05-17 22:18:35,779 INFO recovery.RecoveryService (RecoveryService.java:handle(306)) - DAG completed, dagId=dag_1463523515295_0001_1, queueSize=0
2016-05-17 22:18:35,779 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:handleEvent(151)) - Writing event DAG_FINISHED to history file
2016-05-17 22:18:35,780 INFO history.HistoryEventHandler (HistoryEventHandler.java:handleCriticalEvent(110)) - [HISTORY][DAG:dag_1463523515295_0001_1][Event:DAG_FINISHED]: dagId=dag_1463523515295_0001_1, startTime=1463523515472, finishTime=1463523515777, timeTaken=305, status=SUCCEEDED, diagnostics=, counters=Counters: 55, org.apache.tez.common.counters.DAGCounter, NUM_SUCCEEDED_TASKS=2, TOTAL_LAUNCHED_TASKS=2, RACK_LOCAL_TASKS=1, File System Counters, FILE_BYTES_READ=317, FILE_BYTES_WRITTEN=759, FILE_READ_OPS=0, FILE_LARGE_READ_OPS=0, FILE_WRITE_OPS=0, org.apache.tez.common.counters.TaskCounter, REDUCE_INPUT_GROUPS=4, REDUCE_INPUT_RECORDS=5, COMBINE_INPUT_RECORDS=0, SPILLED_RECORDS=26, NUM_SHUFFLED_INPUTS=1, NUM_SKIPPED_INPUTS=0, NUM_FAILED_SHUFFLE_INPUTS=0, MERGED_MAP_OUTPUTS=1, GC_TIME_MILLIS=110, CPU_MILLISECONDS=0, PHYSICAL_MEMORY_BYTES=0, VIRTUAL_MEMORY_BYTES=0, COMMITTED_HEAP_BYTES=1495269376, INPUT_RECORDS_PROCESSED=0, OUTPUT_RECORDS=18, OUTPUT_BYTES=195, OUTPUT_BYTES_WITH_OVERHEAD=171, OUTPUT_BYTES_PHYSICAL=175, ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=175, ADDITIONAL_SPILL_COUNT=0, SHUFFLE_BYTES=175, SHUFFLE_BYTES_DECOMPRESSED=171, SHUFFLE_BYTES_TO_MEM=0, SHUFFLE_BYTES_TO_DISK=0, SHUFFLE_BYTES_DISK_DIRECT=175, NUM_MEM_TO_DISK_MERGES=0, NUM_DISK_TO_DISK_MERGES=0, SHUFFLE_PHASE_TIME=10, MERGE_PHASE_TIME=12, FIRST_EVENT_RECEIVED=8, LAST_EVENT_RECEIVED=8, Shuffle Errors, BAD_ID=0, CONNECTION=0, IO_ERROR=0, WRONG_LENGTH=0, WRONG_MAP=0, WRONG_REDUCE=0, cascading.flow.SliceCounters, Process_Begin_Time=2927047031418, Process_Duration=10, Process_End_Time=2927047031428, Read_Duration=2, Tuples_Read=18, Tuples_Written=18, Write_Duration=2, cascading.flow.StepCounters, Tuples_Read=13, Tuples_Written=5
2016-05-17 22:18:35,780 INFO impl.DAGImpl (DAGImpl.java:finished(1180)) - DAG: dag_1463523515295_0001_1 finished with state: SUCCEEDED
2016-05-17 22:18:35,781 INFO impl.DAGImpl (DAGImpl.java:handle(973)) - dag_1463523515295_0001_1 transitioned from RUNNING to SUCCEEDED due to event DAG_VERTEX_COMPLETED
2016-05-17 22:18:35,781 INFO launcher.LocalContainerLauncher (LocalContainerLauncher.java:stop(242)) - Ignoring stop request for containerId: container_1463523515295_0001_00_000002
2016-05-17 22:18:35,781 INFO container.AMContainerImpl (AMContainerImpl.java:handle(442)) - AMContainer container_1463523515295_0001_00_000002 transitioned from STOP_REQUESTED to COMPLETED via event C_COMPLETED
2016-05-17 22:18:35 Completed Dag: dag_1463523515295_0001_1
2016-05-17 22:18:35,781 INFO app.DAGAppMaster (DAGAppMaster.java:handle(579)) - DAG completed, dagId=dag_1463523515295_0001_1, dagState=SUCCEEDED
2016-05-17 22:18:35,782 INFO common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(164)) - Redirecting log file based on addend: dag_1463523515295_0001_1_post
2016-05-17 22:18:35,782 WARN common.TezUtilsInternal (TezUtilsInternal.java:updateLoggers(179)) - Not configured with appender named: CLA. Cannot reconfigure logger output
2016-05-17 22:18:35,782 INFO app.DAGAppMaster (DAGAppMaster.java:handle(625)) - Waiting for next DAG to be submitted.
2016-05-17 22:18:35,789 INFO client.TezClient (TezClient.java:stop(495)) - Shutting down Tez Session, sessionName=(1/1) ...m/testFirstNBuffer/firstn, applicationId=application_1463523515295_0001
2016-05-17 22:18:35,791 INFO client.DAGClientHandler (DAGClientHandler.java:shutdownAM(117)) - Received message to shutdown AM
2016-05-17 22:18:35,791 INFO rm.TaskSchedulerEventHandler (TaskSchedulerEventHandler.java:setShouldUnregisterFlag(580)) - TaskScheduler notified that it should unregister from RM
2016-05-17 22:18:35,791 INFO app.DAGAppMaster (DAGAppMaster.java:shutdownTezAM(1088)) - No current running DAG, shutting down the AM
2016-05-17 22:18:35,791 INFO app.DAGAppMaster (DAGAppMaster.java:shutdown(692)) - DAGAppMasterShutdownHandler invoked
2016-05-17 22:18:35,792 INFO app.DAGAppMaster (DAGAppMaster.java:shutdown(701)) - Handling DAGAppMaster shutdown
2016-05-17 22:18:35,792 INFO app.DAGAppMaster (DAGAppMaster.java:run(721)) - Sleeping for 5 seconds before shutting down
2016-05-17 22:18:35,792 INFO history.HistoryEventHandler (HistoryEventHandler.java:serviceStop(79)) - Stopping HistoryEventHandler
2016-05-17 22:18:35,793 INFO recovery.RecoveryService (RecoveryService.java:serviceStop(189)) - Stopping RecoveryService
2016-05-17 22:18:35,794 INFO recovery.RecoveryService (RecoveryService.java:serviceStop(192)) - Handle the remaining events in queue, queue size=0
2016-05-17 22:18:35,794 INFO recovery.RecoveryService (RecoveryService.java:run(166)) - EventQueue take interrupted. Returning
2016-05-17 22:18:35,794 INFO recovery.RecoveryService (RecoveryService.java:serviceStop(214)) - Closing Summary Stream
2016-05-17 22:18:35,794 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:serviceStop(118)) - Stopping SimpleHistoryLoggingService, eventQueueBacklog=0
2016-05-17 22:18:35,794 INFO impl.SimpleHistoryLoggingService (SimpleHistoryLoggingService.java:run(105)) - EventQueue take interrupted. Returning
2016-05-17 22:18:35,799 INFO app.DAGAppMaster (DAGAppMaster.java:run(1773)) - Completed deletion of tez scratch data dir, path=file:/opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.5.x-tez-0.6.x/build/test/tmp/.staging/B5FF3D6B3FEC4DE2963B32969B26C7D0/A34225503819433D86E0612345749768/.tez/application_1463523515295_0001
2016-05-17 22:18:35,800 INFO util.Hadoop18TapUtil (Hadoop18TapUtil.java:cleanTempPath(226)) - deleting temp path /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.5.x-tez-0.6.x/build/test/output/hadoop2-tez/bufferpipesplatform/testFirstNBuffer/firstn/_temporary
2016-05-17 22:18:35,800 INFO flow.Flow (BaseFlow.java:logInfo(1504)) - [test] completed in: 00:00.530
2016-05-17 22:18:35,868 INFO cascading.PlatformTestCase (PlatformTestCase.java:tearDown(189)) - copying to local /opt/buildAgent-8.0.3/work/2f4c45215847827d/apache-2.5.x-tez-0.6.x/build/test/output//hadoop2-tez/bufferpipesplatform/testFirstNBuffer/firstn