2015-02-12 09:42:28,490 INFO platform.PlatformRunner (PlatformRunner.java:getPlatformClassesFromClasspath(187)) - found 1 test platforms from classpath
2015-02-12 09:42:28,523 INFO platform.PlatformRunner (PlatformRunner.java:addPlatform(237)) - installing platform: hadoop2-mr1
2015-02-12 09:42:28,523 INFO platform.PlatformRunner (PlatformRunner.java:addPlatform(238)) - running test: cascading.BasicPipesPlatformTest
2015-02-12 09:42:28,538 INFO hadoop2.Hadoop2MR1Platform (Hadoop2MR1Platform.java:setUp(105)) - not using cluster
2015-02-12 09:42:29,609 WARN util.NativeCodeLoader (NativeCodeLoader.java:<clinit>(62)) - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
2015-02-12 09:42:29,758 INFO util.HadoopUtil (HadoopUtil.java:findMainClass(336)) - using default application jar, may cause class not found exceptions on the cluster
2015-02-12 09:42:29,759 INFO planner.HadoopPlanner (HadoopPlanner.java:initialize(225)) - using application jar: /Users/soberoi/.gradle/caches/modules-2/files-2.1/cascading/cascading-hadoop2-mr1/2.6.3/9ae016e2b8d246b9c1de9168e715d9f1b971aa4a/cascading-hadoop2-mr1-2.6.3.jar
2015-02-12 09:42:29,767 INFO property.AppProps (AppProps.java:getAppID(169)) - using app.id: 93965D71595944ADA399959F3E9FC2E0
2015-02-12 09:42:29,826 INFO Configuration.deprecation (Configuration.java:warnOnceIfDeprecated(1051)) - mapred.used.genericoptionsparser is deprecated. Instead, use mapreduce.client.genericoptionsparser.used
2015-02-12 09:42:29,854 INFO Configuration.deprecation (Configuration.java:warnOnceIfDeprecated(1051)) - mapred.job.tracker is deprecated. Instead, use mapreduce.jobtracker.address
2015-02-12 09:42:29,899 INFO Configuration.deprecation (Configuration.java:warnOnceIfDeprecated(1051)) - mapred.output.compress is deprecated. Instead, use mapreduce.output.fileoutputformat.compress
2015-02-12 09:42:29,933 INFO util.Version (Version.java:printBanner(82)) - Concurrent, Inc - Cascading 2.6.3
2015-02-12 09:42:29,934 INFO flow.Flow (BaseFlow.java:logInfo(1378)) - [test] starting
2015-02-12 09:42:29,935 INFO flow.Flow (BaseFlow.java:logInfo(1378)) - [test] source: Hfs["TextLine[[0:1]->[ALL]]"]["/Users/soberoi/src/compatibility/cascading.compatibility/horton/build/classes/test/data/apache.10.txt"]
2015-02-12 09:42:29,935 INFO flow.Flow (BaseFlow.java:logInfo(1378)) - [test] sink: Hfs["TextLine[['offset', 'line']->[ALL]]"]["/Users/soberoi/src/compatibility/cascading.compatibility/horton/build/test/output/hadoop2-mr1/basicpipesplatform/simplechain"]
2015-02-12 09:42:29,935 INFO flow.Flow (BaseFlow.java:logInfo(1378)) - [test] parallel execution is enabled: false
2015-02-12 09:42:29,936 INFO flow.Flow (BaseFlow.java:logInfo(1378)) - [test] starting jobs: 2
2015-02-12 09:42:29,936 INFO flow.Flow (BaseFlow.java:logInfo(1378)) - [test] allocating threads: 1
2015-02-12 09:42:29,937 INFO flow.FlowStep (BaseFlowStep.java:logInfo(834)) - [test] starting step: (1/2)
2015-02-12 09:42:29,951 INFO Configuration.deprecation (Configuration.java:warnOnceIfDeprecated(1051)) - session.id is deprecated. Instead, use dfs.metrics.session-id
2015-02-12 09:42:29,952 INFO jvm.JvmMetrics (JvmMetrics.java:init(76)) - Initializing JVM Metrics with processName=JobTracker, sessionId=
2015-02-12 09:42:29,966 INFO jvm.JvmMetrics (JvmMetrics.java:init(71)) - Cannot initialize JVM Metrics with processName=JobTracker, sessionId= - already initialized
2015-02-12 09:42:30,131 INFO mapred.FileInputFormat (FileInputFormat.java:listStatus(247)) - Total input paths to process : 1
2015-02-12 09:42:30,157 INFO mapreduce.JobSubmitter (JobSubmitter.java:submitJobInternal(494)) - number of splits:1
2015-02-12 09:42:30,323 INFO mapreduce.JobSubmitter (JobSubmitter.java:printTokens(583)) - Submitting tokens for job: job_local686335487_0001
2015-02-12 09:42:30,449 INFO mapreduce.Job (Job.java:submit(1300)) - The url to track the job: http://localhost:8080/
2015-02-12 09:42:30,450 INFO mapred.LocalJobRunner (LocalJobRunner.java:createOutputCommitter(471)) - OutputCommitter set in config null
2015-02-12 09:42:30,452 INFO mapred.LocalJobRunner (LocalJobRunner.java:createOutputCommitter(489)) - OutputCommitter is org.apache.hadoop.mapred.FileOutputCommitter
2015-02-12 09:42:30,453 INFO flow.FlowStep (BaseFlowStep.java:logInfo(834)) - [test] submitted hadoop job: job_local686335487_0001
2015-02-12 09:42:30,454 INFO flow.FlowStep (BaseFlowStep.java:logInfo(834)) - [test] tracking url: http://localhost:8080/
2015-02-12 09:42:30,479 INFO mapred.LocalJobRunner (LocalJobRunner.java:runTasks(448)) - Waiting for map tasks
2015-02-12 09:42:30,481 INFO mapred.LocalJobRunner (LocalJobRunner.java:run(224)) - Starting task: attempt_local686335487_0001_m_000000_0
2015-02-12 09:42:30,501 INFO util.ProcfsBasedProcessTree (ProcfsBasedProcessTree.java:isAvailable(181)) - ProcfsBasedProcessTree currently is supported only on Linux.
2015-02-12 09:42:30,501 INFO mapred.Task (Task.java:initialize(587)) - Using ResourceCalculatorProcessTree : null
2015-02-12 09:42:30,507 INFO io.MultiInputSplit (MultiInputSplit.java:readFields(161)) - current split input path: file:/Users/soberoi/src/compatibility/cascading.compatibility/horton/build/classes/test/data/apache.10.txt
2015-02-12 09:42:30,507 INFO mapred.MapTask (MapTask.java:updateJobWithSplit(482)) - Processing split: cascading.tap.hadoop.io.MultiInputSplit@21286868
2015-02-12 09:42:30,513 INFO mapred.MapTask (MapTask.java:runOldMapper(436)) - numReduceTasks: 1
2015-02-12 09:42:30,568 INFO mapred.MapTask (MapTask.java:setEquator(1202)) - (EQUATOR) 0 kvi 26214396(104857584)
2015-02-12 09:42:30,568 INFO mapred.MapTask (MapTask.java:init(995)) - mapreduce.task.io.sort.mb: 100
2015-02-12 09:42:30,568 INFO mapred.MapTask (MapTask.java:init(996)) - soft limit at 83886080
2015-02-12 09:42:30,568 INFO mapred.MapTask (MapTask.java:init(997)) - bufstart = 0; bufvoid = 104857600
2015-02-12 09:42:30,569 INFO mapred.MapTask (MapTask.java:init(998)) - kvstart = 26214396; length = 6553600
2015-02-12 09:42:30,587 INFO mapred.MapTask (MapTask.java:createSortingCollector(402)) - Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer
2015-02-12 09:42:30,588 INFO hadoop.FlowMapper (FlowMapper.java:configure(71)) - cascading version: 2.6.3
2015-02-12 09:42:30,588 INFO hadoop.FlowMapper (FlowMapper.java:configure(72)) - child jvm opts: -Xmx200m
2015-02-12 09:42:30,617 INFO Configuration.deprecation (Configuration.java:warnOnceIfDeprecated(1051)) - mapred.task.partition is deprecated. Instead, use mapreduce.task.partition
2015-02-12 09:42:30,627 INFO hadoop.FlowMapper (FlowMapper.java:configure(87)) - sourcing from: Hfs["TextLine[[0:1]->[ALL]]"]["/Users/soberoi/src/compatibility/cascading.compatibility/horton/build/classes/test/data/apache.10.txt"]
2015-02-12 09:42:30,627 INFO hadoop.FlowMapper (FlowMapper.java:configure(90)) - sinking to: GroupBy(test)[by:[{1}:0]]
2015-02-12 09:42:30,636 INFO mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) -
2015-02-12 09:42:30,636 INFO mapred.MapTask (MapTask.java:flush(1457)) - Starting flush of map output
2015-02-12 09:42:30,636 INFO mapred.MapTask (MapTask.java:flush(1475)) - Spilling map output
2015-02-12 09:42:30,637 INFO mapred.MapTask (MapTask.java:flush(1476)) - bufstart = 0; bufend = 210; bufvoid = 104857600
2015-02-12 09:42:30,637 INFO mapred.MapTask (MapTask.java:flush(1478)) - kvstart = 26214396(104857584); kvend = 26214360(104857440); length = 37/6553600
2015-02-12 09:42:30,642 INFO mapred.MapTask (MapTask.java:sortAndSpill(1660)) - Finished spill 0
2015-02-12 09:42:30,645 INFO mapred.Task (Task.java:done(1001)) - Task:attempt_local686335487_0001_m_000000_0 is done. And is in the process of committing
2015-02-12 09:42:30,653 INFO mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) - file:/Users/soberoi/src/compatibility/cascading.compatibility/horton/build/classes/test/data/apache.10.txt:0+1449
2015-02-12 09:42:30,653 INFO mapred.Task (Task.java:sendDone(1121)) - Task 'attempt_local686335487_0001_m_000000_0' done.
2015-02-12 09:42:30,653 INFO mapred.LocalJobRunner (LocalJobRunner.java:run(249)) - Finishing task: attempt_local686335487_0001_m_000000_0
2015-02-12 09:42:30,653 INFO mapred.LocalJobRunner (LocalJobRunner.java:runTasks(456)) - map task executor complete.
2015-02-12 09:42:30,656 INFO mapred.LocalJobRunner (LocalJobRunner.java:runTasks(448)) - Waiting for reduce tasks
2015-02-12 09:42:30,656 INFO mapred.LocalJobRunner (LocalJobRunner.java:run(302)) - Starting task: attempt_local686335487_0001_r_000000_0
2015-02-12 09:42:30,663 INFO util.ProcfsBasedProcessTree (ProcfsBasedProcessTree.java:isAvailable(181)) - ProcfsBasedProcessTree currently is supported only on Linux.
2015-02-12 09:42:30,663 INFO mapred.Task (Task.java:initialize(587)) - Using ResourceCalculatorProcessTree : null
2015-02-12 09:42:30,665 INFO mapred.ReduceTask (ReduceTask.java:run(362)) - Using ShuffleConsumerPlugin: org.apache.hadoop.mapreduce.task.reduce.Shuffle@47c38a82
2015-02-12 09:42:30,674 INFO reduce.MergeManagerImpl (MergeManagerImpl.java:<init>(196)) - MergerManager: memoryLimit=493250144, maxSingleShuffleLimit=123312536, mergeThreshold=325545120, ioSortFactor=10, memToMemMergeOutputsThreshold=10
2015-02-12 09:42:30,676 INFO reduce.EventFetcher (EventFetcher.java:run(61)) - attempt_local686335487_0001_r_000000_0 Thread started: EventFetcher for fetching Map Completion Events
2015-02-12 09:42:30,695 INFO reduce.LocalFetcher (LocalFetcher.java:copyMapOutput(141)) - localfetcher#1 about to shuffle output of map attempt_local686335487_0001_m_000000_0 decomp: 232 len: 236 to MEMORY
2015-02-12 09:42:30,699 INFO reduce.InMemoryMapOutput (InMemoryMapOutput.java:shuffle(100)) - Read 232 bytes from map-output for attempt_local686335487_0001_m_000000_0
2015-02-12 09:42:30,721 INFO reduce.MergeManagerImpl (MergeManagerImpl.java:closeInMemoryFile(314)) - closeInMemoryFile -> map-output of size: 232, inMemoryMapOutputs.size() -> 1, commitMemory -> 0, usedMemory ->232
2015-02-12 09:42:30,722 INFO reduce.EventFetcher (EventFetcher.java:run(76)) - EventFetcher is interrupted.. Returning
2015-02-12 09:42:30,723 INFO mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) - 1 / 1 copied.
2015-02-12 09:42:30,723 INFO reduce.MergeManagerImpl (MergeManagerImpl.java:finalMerge(674)) - finalMerge called with 1 in-memory map-outputs and 0 on-disk map-outputs
2015-02-12 09:42:30,729 INFO mapred.Merger (Merger.java:merge(597)) - Merging 1 sorted segments
2015-02-12 09:42:30,730 INFO mapred.Merger (Merger.java:merge(696)) - Down to the last merge-pass, with 1 segments left of total size: 211 bytes
2015-02-12 09:42:30,731 INFO reduce.MergeManagerImpl (MergeManagerImpl.java:finalMerge(751)) - Merged 1 segments, 232 bytes to disk to satisfy reduce memory limit
2015-02-12 09:42:30,731 INFO reduce.MergeManagerImpl (MergeManagerImpl.java:finalMerge(781)) - Merging 1 files, 236 bytes from disk
2015-02-12 09:42:30,732 INFO reduce.MergeManagerImpl (MergeManagerImpl.java:finalMerge(796)) - Merging 0 segments, 0 bytes from memory into reduce
2015-02-12 09:42:30,732 INFO mapred.Merger (Merger.java:merge(597)) - Merging 1 sorted segments
2015-02-12 09:42:30,733 INFO mapred.Merger (Merger.java:merge(696)) - Down to the last merge-pass, with 1 segments left of total size: 211 bytes
2015-02-12 09:42:30,733 INFO mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) - 1 / 1 copied.
2015-02-12 09:42:30,735 INFO hadoop.FlowReducer (FlowReducer.java:configure(78)) - cascading version: 2.6.3
2015-02-12 09:42:30,735 INFO hadoop.FlowReducer (FlowReducer.java:configure(79)) - child jvm opts: -Xmx200m
2015-02-12 09:42:30,747 INFO hadoop.FlowReducer (FlowReducer.java:configure(97)) - sourcing from: GroupBy(test)[by:[{1}:0]]
2015-02-12 09:42:30,747 INFO hadoop.FlowReducer (FlowReducer.java:configure(100)) - sinking to: TempHfs["SequenceFile[[0, 'count']]"][3767895837/test/]
2015-02-12 09:42:30,772 INFO mapred.Task (Task.java:done(1001)) - Task:attempt_local686335487_0001_r_000000_0 is done. And is in the process of committing
2015-02-12 09:42:30,773 INFO mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) - 1 / 1 copied.
2015-02-12 09:42:30,773 INFO mapred.Task (Task.java:commit(1162)) - Task attempt_local686335487_0001_r_000000_0 is allowed to commit now
2015-02-12 09:42:30,774 INFO output.FileOutputCommitter (FileOutputCommitter.java:commitTask(439)) - Saved output of task 'attempt_local686335487_0001_r_000000_0' to file:/tmp/hadoop-soberoi/3767895837_test_8A8901002FF048898E30502F2A128E96/_temporary/0/task_local686335487_0001_r_000000
2015-02-12 09:42:30,775 INFO mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) - reduce > reduce
2015-02-12 09:42:30,775 INFO mapred.Task (Task.java:sendDone(1121)) - Task 'attempt_local686335487_0001_r_000000_0' done.
2015-02-12 09:42:30,775 INFO mapred.LocalJobRunner (LocalJobRunner.java:run(325)) - Finishing task: attempt_local686335487_0001_r_000000_0
2015-02-12 09:42:30,776 INFO mapred.LocalJobRunner (LocalJobRunner.java:runTasks(456)) - reduce task executor complete.
2015-02-12 09:42:30,808 INFO flow.FlowStep (BaseFlowStep.java:logInfo(834)) - [test] starting step: (2/2) ...pipesplatform/simplechain
2015-02-12 09:42:30,809 INFO jvm.JvmMetrics (JvmMetrics.java:init(71)) - Cannot initialize JVM Metrics with processName=JobTracker, sessionId= - already initialized
2015-02-12 09:42:30,811 INFO jvm.JvmMetrics (JvmMetrics.java:init(71)) - Cannot initialize JVM Metrics with processName=JobTracker, sessionId= - already initialized
2015-02-12 09:42:30,849 INFO mapred.FileInputFormat (FileInputFormat.java:listStatus(247)) - Total input paths to process : 1
2015-02-12 09:42:30,868 INFO mapreduce.JobSubmitter (JobSubmitter.java:submitJobInternal(494)) - number of splits:1
2015-02-12 09:42:30,880 INFO Configuration.deprecation (Configuration.java:warnOnceIfDeprecated(1051)) - mapred.used.genericoptionsparser is deprecated. Instead, use mapreduce.client.genericoptionsparser.used
2015-02-12 09:42:30,881 INFO Configuration.deprecation (Configuration.java:warnOnceIfDeprecated(1051)) - mapred.job.tracker is deprecated. Instead, use mapreduce.jobtracker.address
2015-02-12 09:42:30,908 INFO mapreduce.JobSubmitter (JobSubmitter.java:printTokens(583)) - Submitting tokens for job: job_local985118429_0002
2015-02-12 09:42:30,984 INFO mapreduce.Job (Job.java:submit(1300)) - The url to track the job: http://localhost:8080/
2015-02-12 09:42:30,984 INFO mapred.LocalJobRunner (LocalJobRunner.java:createOutputCommitter(471)) - OutputCommitter set in config null
2015-02-12 09:42:30,985 INFO mapred.LocalJobRunner (LocalJobRunner.java:createOutputCommitter(489)) - OutputCommitter is org.apache.hadoop.mapred.FileOutputCommitter
2015-02-12 09:42:30,989 INFO mapred.LocalJobRunner (LocalJobRunner.java:runTasks(448)) - Waiting for map tasks
2015-02-12 09:42:30,989 INFO mapred.LocalJobRunner (LocalJobRunner.java:run(224)) - Starting task: attempt_local985118429_0002_m_000000_0
2015-02-12 09:42:30,992 INFO flow.FlowStep (BaseFlowStep.java:logInfo(834)) - [test] submitted hadoop job: job_local985118429_0002
2015-02-12 09:42:30,993 INFO util.ProcfsBasedProcessTree (ProcfsBasedProcessTree.java:isAvailable(181)) - ProcfsBasedProcessTree currently is supported only on Linux.
2015-02-12 09:42:30,993 INFO mapred.Task (Task.java:initialize(587)) - Using ResourceCalculatorProcessTree : null
2015-02-12 09:42:30,994 INFO flow.FlowStep (BaseFlowStep.java:logInfo(834)) - [test] tracking url: http://localhost:8080/
2015-02-12 09:42:30,995 INFO io.MultiInputSplit (MultiInputSplit.java:readFields(161)) - current split input path: file:/tmp/hadoop-soberoi/3767895837_test_8A8901002FF048898E30502F2A128E96/part-00000
2015-02-12 09:42:30,995 INFO mapred.MapTask (MapTask.java:updateJobWithSplit(482)) - Processing split: cascading.tap.hadoop.io.MultiInputSplit@39301cdb
2015-02-12 09:42:31,011 INFO mapred.MapTask (MapTask.java:runOldMapper(436)) - numReduceTasks: 1
2015-02-12 09:42:31,042 INFO mapred.MapTask (MapTask.java:setEquator(1202)) - (EQUATOR) 0 kvi 26214396(104857584)
2015-02-12 09:42:31,042 INFO mapred.MapTask (MapTask.java:init(995)) - mapreduce.task.io.sort.mb: 100
2015-02-12 09:42:31,042 INFO mapred.MapTask (MapTask.java:init(996)) - soft limit at 83886080
2015-02-12 09:42:31,042 INFO mapred.MapTask (MapTask.java:init(997)) - bufstart = 0; bufvoid = 104857600
2015-02-12 09:42:31,043 INFO mapred.MapTask (MapTask.java:init(998)) - kvstart = 26214396; length = 6553600
2015-02-12 09:42:31,044 INFO mapred.MapTask (MapTask.java:createSortingCollector(402)) - Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer
2015-02-12 09:42:31,044 INFO hadoop.FlowMapper (FlowMapper.java:configure(71)) - cascading version: 2.6.3
2015-02-12 09:42:31,044 INFO hadoop.FlowMapper (FlowMapper.java:configure(72)) - child jvm opts: -Xmx200m
2015-02-12 09:42:31,050 INFO hadoop.FlowMapper (FlowMapper.java:configure(87)) - sourcing from: TempHfs["SequenceFile[[0, 'count']]"][3767895837/test/]
2015-02-12 09:42:31,050 INFO hadoop.FlowMapper (FlowMapper.java:configure(90)) - sinking to: GroupBy(test)[by:[{1}:0]]
2015-02-12 09:42:31,052 INFO mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) -
2015-02-12 09:42:31,052 INFO mapred.MapTask (MapTask.java:flush(1457)) - Starting flush of map output
2015-02-12 09:42:31,052 INFO mapred.MapTask (MapTask.java:flush(1475)) - Spilling map output
2015-02-12 09:42:31,053 INFO mapred.MapTask (MapTask.java:flush(1476)) - bufstart = 0; bufend = 184; bufvoid = 104857600
2015-02-12 09:42:31,053 INFO mapred.MapTask (MapTask.java:flush(1478)) - kvstart = 26214396(104857584); kvend = 26214368(104857472); length = 29/6553600
2015-02-12 09:42:31,055 INFO mapred.MapTask (MapTask.java:sortAndSpill(1660)) - Finished spill 0
2015-02-12 09:42:31,057 INFO mapred.Task (Task.java:done(1001)) - Task:attempt_local985118429_0002_m_000000_0 is done. And is in the process of committing
2015-02-12 09:42:31,058 INFO mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) - file:/tmp/hadoop-soberoi/3767895837_test_8A8901002FF048898E30502F2A128E96/part-00000:0+310
2015-02-12 09:42:31,058 INFO mapred.Task (Task.java:sendDone(1121)) - Task 'attempt_local985118429_0002_m_000000_0' done.
2015-02-12 09:42:31,059 INFO mapred.LocalJobRunner (LocalJobRunner.java:run(249)) - Finishing task: attempt_local985118429_0002_m_000000_0
2015-02-12 09:42:31,059 INFO mapred.LocalJobRunner (LocalJobRunner.java:runTasks(456)) - map task executor complete.
2015-02-12 09:42:31,059 INFO mapred.LocalJobRunner (LocalJobRunner.java:runTasks(448)) - Waiting for reduce tasks
2015-02-12 09:42:31,060 INFO mapred.LocalJobRunner (LocalJobRunner.java:run(302)) - Starting task: attempt_local985118429_0002_r_000000_0
2015-02-12 09:42:31,061 INFO util.ProcfsBasedProcessTree (ProcfsBasedProcessTree.java:isAvailable(181)) - ProcfsBasedProcessTree currently is supported only on Linux.
2015-02-12 09:42:31,061 INFO mapred.Task (Task.java:initialize(587)) - Using ResourceCalculatorProcessTree : null
2015-02-12 09:42:31,061 INFO mapred.ReduceTask (ReduceTask.java:run(362)) - Using ShuffleConsumerPlugin: org.apache.hadoop.mapreduce.task.reduce.Shuffle@126b09c0
2015-02-12 09:42:31,062 INFO reduce.MergeManagerImpl (MergeManagerImpl.java:<init>(196)) - MergerManager: memoryLimit=493250144, maxSingleShuffleLimit=123312536, mergeThreshold=325545120, ioSortFactor=10, memToMemMergeOutputsThreshold=10
2015-02-12 09:42:31,063 INFO reduce.EventFetcher (EventFetcher.java:run(61)) - attempt_local985118429_0002_r_000000_0 Thread started: EventFetcher for fetching Map Completion Events
2015-02-12 09:42:31,065 INFO reduce.LocalFetcher (LocalFetcher.java:copyMapOutput(141)) - localfetcher#2 about to shuffle output of map attempt_local985118429_0002_m_000000_0 decomp: 202 len: 206 to MEMORY
2015-02-12 09:42:31,066 INFO reduce.InMemoryMapOutput (InMemoryMapOutput.java:shuffle(100)) - Read 202 bytes from map-output for attempt_local985118429_0002_m_000000_0
2015-02-12 09:42:31,066 INFO reduce.MergeManagerImpl (MergeManagerImpl.java:closeInMemoryFile(314)) - closeInMemoryFile -> map-output of size: 202, inMemoryMapOutputs.size() -> 1, commitMemory -> 0, usedMemory ->202
2015-02-12 09:42:31,067 INFO reduce.EventFetcher (EventFetcher.java:run(76)) - EventFetcher is interrupted.. Returning
2015-02-12 09:42:31,068 INFO mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) - 1 / 1 copied.
2015-02-12 09:42:31,068 INFO reduce.MergeManagerImpl (MergeManagerImpl.java:finalMerge(674)) - finalMerge called with 1 in-memory map-outputs and 0 on-disk map-outputs
2015-02-12 09:42:31,069 INFO mapred.Merger (Merger.java:merge(597)) - Merging 1 sorted segments
2015-02-12 09:42:31,070 INFO mapred.Merger (Merger.java:merge(696)) - Down to the last merge-pass, with 1 segments left of total size: 181 bytes
2015-02-12 09:42:31,071 INFO reduce.MergeManagerImpl (MergeManagerImpl.java:finalMerge(751)) - Merged 1 segments, 202 bytes to disk to satisfy reduce memory limit
2015-02-12 09:42:31,071 INFO reduce.MergeManagerImpl (MergeManagerImpl.java:finalMerge(781)) - Merging 1 files, 206 bytes from disk
2015-02-12 09:42:31,071 INFO reduce.MergeManagerImpl (MergeManagerImpl.java:finalMerge(796)) - Merging 0 segments, 0 bytes from memory into reduce
2015-02-12 09:42:31,071 INFO mapred.Merger (Merger.java:merge(597)) - Merging 1 sorted segments
2015-02-12 09:42:31,072 INFO mapred.Merger (Merger.java:merge(696)) - Down to the last merge-pass, with 1 segments left of total size: 181 bytes
2015-02-12 09:42:31,072 INFO mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) - 1 / 1 copied.
2015-02-12 09:42:31,073 INFO hadoop.FlowReducer (FlowReducer.java:configure(78)) - cascading version: 2.6.3
2015-02-12 09:42:31,073 INFO hadoop.FlowReducer (FlowReducer.java:configure(79)) - child jvm opts: -Xmx200m
2015-02-12 09:42:31,078 INFO hadoop.FlowReducer (FlowReducer.java:configure(97)) - sourcing from: GroupBy(test)[by:[{1}:0]]
2015-02-12 09:42:31,079 INFO hadoop.FlowReducer (FlowReducer.java:configure(100)) - sinking to: Hfs["TextLine[['offset', 'line']->[ALL]]"]["/Users/soberoi/src/compatibility/cascading.compatibility/horton/build/test/output/hadoop2-mr1/basicpipesplatform/simplechain"]
2015-02-12 09:42:31,096 INFO mapred.Task (Task.java:done(1001)) - Task:attempt_local985118429_0002_r_000000_0 is done. And is in the process of committing
2015-02-12 09:42:31,097 INFO mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) - 1 / 1 copied.
2015-02-12 09:42:31,097 INFO mapred.Task (Task.java:commit(1162)) - Task attempt_local985118429_0002_r_000000_0 is allowed to commit now
2015-02-12 09:42:31,098 INFO output.FileOutputCommitter (FileOutputCommitter.java:commitTask(439)) - Saved output of task 'attempt_local985118429_0002_r_000000_0' to file:/Users/soberoi/src/compatibility/cascading.compatibility/horton/build/test/output/hadoop2-mr1/basicpipesplatform/simplechain/_temporary/0/task_local985118429_0002_r_000000
2015-02-12 09:42:31,099 INFO mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) - reduce > reduce
2015-02-12 09:42:31,099 INFO mapred.Task (Task.java:sendDone(1121)) - Task 'attempt_local985118429_0002_r_000000_0' done.
2015-02-12 09:42:31,099 INFO mapred.LocalJobRunner (LocalJobRunner.java:run(325)) - Finishing task: attempt_local985118429_0002_r_000000_0
2015-02-12 09:42:31,099 INFO mapred.LocalJobRunner (LocalJobRunner.java:runTasks(456)) - reduce task executor complete.
2015-02-12 09:42:31,122 INFO util.Hadoop18TapUtil (Hadoop18TapUtil.java:cleanTempPath(219)) - deleting temp path /Users/soberoi/src/compatibility/cascading.compatibility/horton/build/test/output/hadoop2-mr1/basicpipesplatform/simplechain/_temporary
2015-02-12 09:42:31,153 INFO mapred.FileInputFormat (FileInputFormat.java:listStatus(247)) - Total input paths to process : 1
2015-02-12 09:42:31,154 INFO cascading.PlatformTestCase (PlatformTestCase.java:tearDown(209)) - copying to local /Users/soberoi/src/compatibility/cascading.compatibility/horton/build/test/output//hadoop2-mr1/basicpipesplatform/simplechain
2015-02-12 09:42:31,174 INFO util.HadoopUtil (HadoopUtil.java:findMainClass(336)) - using default application jar, may cause class not found exceptions on the cluster
2015-02-12 09:42:31,175 INFO planner.HadoopPlanner (HadoopPlanner.java:initialize(225)) - using application jar: /Users/soberoi/.gradle/caches/modules-2/files-2.1/cascading/cascading-hadoop2-mr1/2.6.3/9ae016e2b8d246b9c1de9168e715d9f1b971aa4a/cascading-hadoop2-mr1-2.6.3.jar
2015-02-12 09:42:31,184 INFO Configuration.deprecation (Configuration.java:warnOnceIfDeprecated(1051)) - mapred.used.genericoptionsparser is deprecated. Instead, use mapreduce.client.genericoptionsparser.used
2015-02-12 09:42:31,187 INFO Configuration.deprecation (Configuration.java:warnOnceIfDeprecated(1051)) - mapred.job.tracker is deprecated. Instead, use mapreduce.jobtracker.address
2015-02-12 09:42:31,188 INFO Configuration.deprecation (Configuration.java:warnOnceIfDeprecated(1051)) - mapred.output.compress is deprecated. Instead, use mapreduce.output.fileoutputformat.compress
2015-02-12 09:42:31,193 INFO flow.Flow (BaseFlow.java:logInfo(1378)) - [count] starting
2015-02-12 09:42:31,193 INFO flow.Flow (BaseFlow.java:logInfo(1378)) - [count] source: Hfs["TextLine[[0:1]->[ALL]]"]["/Users/soberoi/src/compatibility/cascading.compatibility/horton/build/classes/test/data/ips.20.txt"]
2015-02-12 09:42:31,194 INFO flow.Flow (BaseFlow.java:logInfo(1378)) - [count] sink: Hfs["TextLine[[0]->[ALL]]"]["/Users/soberoi/src/compatibility/cascading.compatibility/horton/build/test/output/hadoop2-mr1/basicpipesplatform/count"]
2015-02-12 09:42:31,194 INFO flow.Flow (BaseFlow.java:logInfo(1378)) - [count] parallel execution is enabled: false
2015-02-12 09:42:31,194 INFO flow.Flow (BaseFlow.java:logInfo(1378)) - [count] starting jobs: 1
2015-02-12 09:42:31,194 INFO flow.Flow (BaseFlow.java:logInfo(1378)) - [count] allocating threads: 1
2015-02-12 09:42:31,194 INFO flow.FlowStep (BaseFlowStep.java:logInfo(834)) - [count] starting step: (1/1) .../basicpipesplatform/count
2015-02-12 09:42:31,195 INFO jvm.JvmMetrics (JvmMetrics.java:init(71)) - Cannot initialize JVM Metrics with processName=JobTracker, sessionId= - already initialized
2015-02-12 09:42:31,196 INFO jvm.JvmMetrics (JvmMetrics.java:init(71)) - Cannot initialize JVM Metrics with processName=JobTracker, sessionId= - already initialized
2015-02-12 09:42:31,218 INFO mapred.FileInputFormat (FileInputFormat.java:listStatus(247)) - Total input paths to process : 1
2015-02-12 09:42:31,235 INFO mapreduce.JobSubmitter (JobSubmitter.java:submitJobInternal(494)) - number of splits:1
2015-02-12 09:42:31,290 INFO mapreduce.JobSubmitter (JobSubmitter.java:printTokens(583)) - Submitting tokens for job: job_local730952180_0003
2015-02-12 09:42:31,373 INFO mapreduce.Job (Job.java:submit(1300)) - The url to track the job: http://localhost:8080/
2015-02-12 09:42:31,373 INFO mapred.LocalJobRunner (LocalJobRunner.java:createOutputCommitter(471)) - OutputCommitter set in config null
2015-02-12 09:42:31,373 INFO flow.FlowStep (BaseFlowStep.java:logInfo(834)) - [count] submitted hadoop job: job_local730952180_0003
2015-02-12 09:42:31,373 INFO mapred.LocalJobRunner (LocalJobRunner.java:createOutputCommitter(489)) - OutputCommitter is org.apache.hadoop.mapred.FileOutputCommitter
2015-02-12 09:42:31,373 INFO flow.FlowStep (BaseFlowStep.java:logInfo(834)) - [count] tracking url: http://localhost:8080/
2015-02-12 09:42:31,375 INFO mapred.LocalJobRunner (LocalJobRunner.java:runTasks(448)) - Waiting for map tasks
2015-02-12 09:42:31,375 INFO mapred.LocalJobRunner (LocalJobRunner.java:run(224)) - Starting task: attempt_local730952180_0003_m_000000_0
2015-02-12 09:42:31,377 INFO util.ProcfsBasedProcessTree (ProcfsBasedProcessTree.java:isAvailable(181)) - ProcfsBasedProcessTree currently is supported only on Linux.
2015-02-12 09:42:31,377 INFO mapred.Task (Task.java:initialize(587)) - Using ResourceCalculatorProcessTree : null
2015-02-12 09:42:31,379 INFO io.MultiInputSplit (MultiInputSplit.java:readFields(161)) - current split input path: file:/Users/soberoi/src/compatibility/cascading.compatibility/horton/build/classes/test/data/ips.20.txt
2015-02-12 09:42:31,379 INFO mapred.MapTask (MapTask.java:updateJobWithSplit(482)) - Processing split: cascading.tap.hadoop.io.MultiInputSplit@2906b654
2015-02-12 09:42:31,381 INFO mapred.MapTask (MapTask.java:runOldMapper(436)) - numReduceTasks: 1
2015-02-12 09:42:31,444 INFO mapred.MapTask (MapTask.java:setEquator(1202)) - (EQUATOR) 0 kvi 26214396(104857584)
2015-02-12 09:42:31,444 INFO mapred.MapTask (MapTask.java:init(995)) - mapreduce.task.io.sort.mb: 100
2015-02-12 09:42:31,444 INFO mapred.MapTask (MapTask.java:init(996)) - soft limit at 83886080
2015-02-12 09:42:31,444 INFO mapred.MapTask (MapTask.java:init(997)) - bufstart = 0; bufvoid = 104857600
2015-02-12 09:42:31,444 INFO mapred.MapTask (MapTask.java:init(998)) - kvstart = 26214396; length = 6553600
2015-02-12 09:42:31,446 INFO mapred.MapTask (MapTask.java:createSortingCollector(402)) - Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer
2015-02-12 09:42:31,446 INFO hadoop.FlowMapper (FlowMapper.java:configure(71)) - cascading version: 2.6.3
2015-02-12 09:42:31,447 INFO hadoop.FlowMapper (FlowMapper.java:configure(72)) - child jvm opts: -Xmx200m
2015-02-12 09:42:31,453 INFO hadoop.FlowMapper (FlowMapper.java:configure(87)) - sourcing from: Hfs["TextLine[[0:1]->[ALL]]"]["/Users/soberoi/src/compatibility/cascading.compatibility/horton/build/classes/test/data/ips.20.txt"]
2015-02-12 09:42:31,454 INFO hadoop.FlowMapper (FlowMapper.java:configure(90)) - sinking to: GroupBy(count)[by:[{1}:1]]
2015-02-12 09:42:31,457 INFO mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) -
2015-02-12 09:42:31,457 INFO mapred.MapTask (MapTask.java:flush(1457)) - Starting flush of map output
2015-02-12 09:42:31,457 INFO mapred.MapTask (MapTask.java:flush(1475)) - Spilling map output
2015-02-12 09:42:31,457 INFO mapred.MapTask (MapTask.java:flush(1476)) - bufstart = 0; bufend = 466; bufvoid = 104857600
2015-02-12 09:42:31,457 INFO mapred.MapTask (MapTask.java:flush(1478)) - kvstart = 26214396(104857584); kvend = 26214320(104857280); length = 77/6553600
2015-02-12 09:42:31,460 INFO mapred.MapTask (MapTask.java:sortAndSpill(1660)) - Finished spill 0
2015-02-12 09:42:31,462 INFO mapred.Task (Task.java:done(1001)) - Task:attempt_local730952180_0003_m_000000_0 is done. And is in the process of committing
2015-02-12 09:42:31,464 INFO mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) - file:/Users/soberoi/src/compatibility/cascading.compatibility/horton/build/classes/test/data/ips.20.txt:0+275
2015-02-12 09:42:31,464 INFO mapred.Task (Task.java:sendDone(1121)) - Task 'attempt_local730952180_0003_m_000000_0' done.
2015-02-12 09:42:31,464 INFO mapred.LocalJobRunner (LocalJobRunner.java:run(249)) - Finishing task: attempt_local730952180_0003_m_000000_0
2015-02-12 09:42:31,464 INFO mapred.LocalJobRunner (LocalJobRunner.java:runTasks(456)) - map task executor complete.
2015-02-12 09:42:31,465 INFO mapred.LocalJobRunner (LocalJobRunner.java:runTasks(448)) - Waiting for reduce tasks
2015-02-12 09:42:31,465 INFO mapred.LocalJobRunner (LocalJobRunner.java:run(302)) - Starting task: attempt_local730952180_0003_r_000000_0
2015-02-12 09:42:31,466 INFO util.ProcfsBasedProcessTree (ProcfsBasedProcessTree.java:isAvailable(181)) - ProcfsBasedProcessTree currently is supported only on Linux.
2015-02-12 09:42:31,467 INFO mapred.Task (Task.java:initialize(587)) - Using ResourceCalculatorProcessTree : null
2015-02-12 09:42:31,467 INFO mapred.ReduceTask (ReduceTask.java:run(362)) - Using ShuffleConsumerPlugin: org.apache.hadoop.mapreduce.task.reduce.Shuffle@76df4d97
2015-02-12 09:42:31,467 INFO reduce.MergeManagerImpl (MergeManagerImpl.java:<init>(196)) - MergerManager: memoryLimit=493250144, maxSingleShuffleLimit=123312536, mergeThreshold=325545120, ioSortFactor=10, memToMemMergeOutputsThreshold=10
2015-02-12 09:42:31,468 INFO reduce.EventFetcher (EventFetcher.java:run(61)) - attempt_local730952180_0003_r_000000_0 Thread started: EventFetcher for fetching Map Completion Events
2015-02-12 09:42:31,469 INFO reduce.LocalFetcher (LocalFetcher.java:copyMapOutput(141)) - localfetcher#3 about to shuffle output of map attempt_local730952180_0003_m_000000_0 decomp: 508 len: 512 to MEMORY
2015-02-12 09:42:31,470 INFO reduce.InMemoryMapOutput (InMemoryMapOutput.java:shuffle(100)) - Read 508 bytes from map-output for attempt_local730952180_0003_m_000000_0
2015-02-12 09:42:31,470 INFO reduce.MergeManagerImpl (MergeManagerImpl.java:closeInMemoryFile(314)) - closeInMemoryFile -> map-output of size: 508, inMemoryMapOutputs.size() -> 1, commitMemory -> 0, usedMemory ->508
2015-02-12 09:42:31,470 INFO reduce.EventFetcher (EventFetcher.java:run(76)) - EventFetcher is interrupted.. Returning
2015-02-12 09:42:31,471 INFO mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) - 1 / 1 copied.
2015-02-12 09:42:31,471 INFO reduce.MergeManagerImpl (MergeManagerImpl.java:finalMerge(674)) - finalMerge called with 1 in-memory map-outputs and 0 on-disk map-outputs
2015-02-12 09:42:31,472 INFO mapred.Merger (Merger.java:merge(597)) - Merging 1 sorted segments
2015-02-12 09:42:31,473 INFO mapred.Merger (Merger.java:merge(696)) - Down to the last merge-pass, with 1 segments left of total size: 487 bytes
2015-02-12 09:42:31,474 INFO reduce.MergeManagerImpl (MergeManagerImpl.java:finalMerge(751)) - Merged 1 segments, 508 bytes to disk to satisfy reduce memory limit
2015-02-12 09:42:31,474 INFO reduce.MergeManagerImpl (MergeManagerImpl.java:finalMerge(781)) - Merging 1 files, 512 bytes from disk
2015-02-12 09:42:31,474 INFO reduce.MergeManagerImpl (MergeManagerImpl.java:finalMerge(796)) - Merging 0 segments, 0 bytes from memory into reduce
2015-02-12 09:42:31,474 INFO mapred.Merger (Merger.java:merge(597)) - Merging 1 sorted segments
2015-02-12 09:42:31,474 INFO mapred.Merger (Merger.java:merge(696)) - Down to the last merge-pass, with 1 segments left of total size: 487 bytes
2015-02-12 09:42:31,475 INFO mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) - 1 / 1 copied.
2015-02-12 09:42:31,475 INFO hadoop.FlowReducer (FlowReducer.java:configure(78)) - cascading version: 2.6.3
2015-02-12 09:42:31,476 INFO hadoop.FlowReducer (FlowReducer.java:configure(79)) - child jvm opts: -Xmx200m
2015-02-12 09:42:31,480 INFO hadoop.FlowReducer (FlowReducer.java:configure(97)) - sourcing from: GroupBy(count)[by:[{1}:1]]
2015-02-12 09:42:31,481 INFO hadoop.FlowReducer (FlowReducer.java:configure(100)) - sinking to: Hfs["TextLine[[0]->[ALL]]"]["/Users/soberoi/src/compatibility/cascading.compatibility/horton/build/test/output/hadoop2-mr1/basicpipesplatform/count"]
2015-02-12 09:42:31,491 INFO mapred.Task (Task.java:done(1001)) - Task:attempt_local730952180_0003_r_000000_0 is done. And is in the process of committing
2015-02-12 09:42:31,493 INFO mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) - 1 / 1 copied.
2015-02-12 09:42:31,493 INFO mapred.Task (Task.java:commit(1162)) - Task attempt_local730952180_0003_r_000000_0 is allowed to commit now
2015-02-12 09:42:31,494 INFO output.FileOutputCommitter (FileOutputCommitter.java:commitTask(439)) - Saved output of task 'attempt_local730952180_0003_r_000000_0' to file:/Users/soberoi/src/compatibility/cascading.compatibility/horton/build/test/output/hadoop2-mr1/basicpipesplatform/count/_temporary/0/task_local730952180_0003_r_000000
2015-02-12 09:42:31,495 INFO mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) - reduce > reduce
2015-02-12 09:42:31,495 INFO mapred.Task (Task.java:sendDone(1121)) - Task 'attempt_local730952180_0003_r_000000_0' done.
2015-02-12 09:42:31,495 INFO mapred.LocalJobRunner (LocalJobRunner.java:run(325)) - Finishing task: attempt_local730952180_0003_r_000000_0
2015-02-12 09:42:31,496 INFO mapred.LocalJobRunner (LocalJobRunner.java:runTasks(456)) - reduce task executor complete.
2015-02-12 09:42:31,513 INFO util.Hadoop18TapUtil (Hadoop18TapUtil.java:cleanTempPath(219)) - deleting temp path /Users/soberoi/src/compatibility/cascading.compatibility/horton/build/test/output/hadoop2-mr1/basicpipesplatform/count/_temporary
2015-02-12 09:42:31,544 INFO mapred.FileInputFormat (FileInputFormat.java:listStatus(247)) - Total input paths to process : 1
2015-02-12 09:42:31,572 INFO mapred.FileInputFormat (FileInputFormat.java:listStatus(247)) - Total input paths to process : 1
2015-02-12 09:42:31,574 INFO cascading.PlatformTestCase (PlatformTestCase.java:tearDown(209)) - copying to local /Users/soberoi/src/compatibility/cascading.compatibility/horton/build/test/output//hadoop2-mr1/basicpipesplatform/count
2015-02-12 09:42:31,591 INFO util.HadoopUtil (HadoopUtil.java:findMainClass(336)) - using default application jar, may cause class not found exceptions on the cluster
2015-02-12 09:42:31,591 INFO planner.HadoopPlanner (HadoopPlanner.java:initialize(225)) - using application jar: /Users/soberoi/.gradle/caches/modules-2/files-2.1/cascading/cascading-hadoop2-mr1/2.6.3/9ae016e2b8d246b9c1de9168e715d9f1b971aa4a/cascading-hadoop2-mr1-2.6.3.jar
2015-02-12 09:42:31,598 INFO Configuration.deprecation (Configuration.java:warnOnceIfDeprecated(1051)) - mapred.used.genericoptionsparser is deprecated. Instead, use mapreduce.client.genericoptionsparser.used
2015-02-12 09:42:31,601 INFO Configuration.deprecation (Configuration.java:warnOnceIfDeprecated(1051)) - mapred.job.tracker is deprecated. Instead, use mapreduce.jobtracker.address
2015-02-12 09:42:31,602 INFO Configuration.deprecation (Configuration.java:warnOnceIfDeprecated(1051)) - mapred.output.compress is deprecated. Instead, use mapreduce.output.fileoutputformat.compress
2015-02-12 09:42:31,607 INFO flow.Flow (BaseFlow.java:logInfo(1378)) - [test] starting
2015-02-12 09:42:31,608 INFO flow.Flow (BaseFlow.java:logInfo(1378)) - [test] source: Hfs["TextLine[[0:1]->[ALL]]"]["/Users/soberoi/src/compatibility/cascading.compatibility/horton/build/classes/test/data/apache.10.txt"]
2015-02-12 09:42:31,608 INFO flow.Flow (BaseFlow.java:logInfo(1378)) - [test] sink: Hfs["TextLine[['offset', 'line']->[ALL]]"]["/Users/soberoi/src/compatibility/cascading.compatibility/horton/build/test/output/hadoop2-mr1/basicpipesplatform/swap"]
2015-02-12 09:42:31,608 INFO flow.Flow (BaseFlow.java:logInfo(1378)) - [test] parallel execution is enabled: false
2015-02-12 09:42:31,608 INFO flow.Flow (BaseFlow.java:logInfo(1378)) - [test] starting jobs: 1
2015-02-12 09:42:31,608 INFO flow.Flow (BaseFlow.java:logInfo(1378)) - [test] allocating threads: 1
2015-02-12 09:42:31,609 INFO flow.FlowStep (BaseFlowStep.java:logInfo(834)) - [test] starting step: (1/1) ...1/basicpipesplatform/swap
2015-02-12 09:42:31,609 INFO jvm.JvmMetrics (JvmMetrics.java:init(71)) - Cannot initialize JVM Metrics with processName=JobTracker, sessionId= - already initialized
2015-02-12 09:42:31,611 INFO jvm.JvmMetrics (JvmMetrics.java:init(71)) - Cannot initialize JVM Metrics with processName=JobTracker, sessionId= - already initialized
2015-02-12 09:42:31,638 INFO mapred.FileInputFormat (FileInputFormat.java:listStatus(247)) - Total input paths to process : 1
2015-02-12 09:42:31,657 INFO mapreduce.JobSubmitter (JobSubmitter.java:submitJobInternal(494)) - number of splits:1
2015-02-12 09:42:31,678 INFO mapreduce.JobSubmitter (JobSubmitter.java:printTokens(583)) - Submitting tokens for job: job_local517689383_0004
2015-02-12 09:42:31,733 INFO mapreduce.Job (Job.java:submit(1300)) - The url to track the job: http://localhost:8080/
2015-02-12 09:42:31,733 INFO mapred.LocalJobRunner (LocalJobRunner.java:createOutputCommitter(471)) - OutputCommitter set in config null
2015-02-12 09:42:31,733 INFO mapred.LocalJobRunner (LocalJobRunner.java:createOutputCommitter(489)) - OutputCommitter is org.apache.hadoop.mapred.FileOutputCommitter
2015-02-12 09:42:31,733 INFO flow.FlowStep (BaseFlowStep.java:logInfo(834)) - [test] submitted hadoop job: job_local517689383_0004
2015-02-12 09:42:31,733 INFO flow.FlowStep (BaseFlowStep.java:logInfo(834)) - [test] tracking url: http://localhost:8080/
2015-02-12 09:42:31,735 INFO mapred.LocalJobRunner (LocalJobRunner.java:runTasks(448)) - Waiting for map tasks
2015-02-12 09:42:31,736 INFO mapred.LocalJobRunner (LocalJobRunner.java:run(224)) - Starting task: attempt_local517689383_0004_m_000000_0
2015-02-12 09:42:31,737 INFO util.ProcfsBasedProcessTree (ProcfsBasedProcessTree.java:isAvailable(181)) - ProcfsBasedProcessTree currently is supported only on Linux.
2015-02-12 09:42:31,737 INFO mapred.Task (Task.java:initialize(587)) - Using ResourceCalculatorProcessTree : null
2015-02-12 09:42:31,738 INFO io.MultiInputSplit (MultiInputSplit.java:readFields(161)) - current split input path: file:/Users/soberoi/src/compatibility/cascading.compatibility/horton/build/classes/test/data/apache.10.txt
2015-02-12 09:42:31,739 INFO mapred.MapTask (MapTask.java:updateJobWithSplit(482)) - Processing split: cascading.tap.hadoop.io.MultiInputSplit@7ae20eb
2015-02-12 09:42:31,739 INFO mapred.MapTask (MapTask.java:runOldMapper(436)) - numReduceTasks: 0
2015-02-12 09:42:31,746 INFO hadoop.FlowMapper (FlowMapper.java:configure(71)) - cascading version: 2.6.3
2015-02-12 09:42:31,746 INFO hadoop.FlowMapper (FlowMapper.java:configure(72)) - child jvm opts: -Xmx200m
2015-02-12 09:42:31,750 INFO hadoop.FlowMapper (FlowMapper.java:configure(87)) - sourcing from: Hfs["TextLine[[0:1]->[ALL]]"]["/Users/soberoi/src/compatibility/cascading.compatibility/horton/build/classes/test/data/apache.10.txt"]
2015-02-12 09:42:31,750 INFO hadoop.FlowMapper (FlowMapper.java:configure(90)) - sinking to: Hfs["TextLine[['offset', 'line']->[ALL]]"]["/Users/soberoi/src/compatibility/cascading.compatibility/horton/build/test/output/hadoop2-mr1/basicpipesplatform/swap"]
2015-02-12 09:42:31,753 INFO mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) -
2015-02-12 09:42:31,753 INFO mapred.Task (Task.java:done(1001)) - Task:attempt_local517689383_0004_m_000000_0 is done. And is in the process of committing
2015-02-12 09:42:31,754 INFO mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) -
2015-02-12 09:42:31,754 INFO mapred.Task (Task.java:commit(1162)) - Task attempt_local517689383_0004_m_000000_0 is allowed to commit now
2015-02-12 09:42:31,755 INFO output.FileOutputCommitter (FileOutputCommitter.java:commitTask(439)) - Saved output of task 'attempt_local517689383_0004_m_000000_0' to file:/Users/soberoi/src/compatibility/cascading.compatibility/horton/build/test/output/hadoop2-mr1/basicpipesplatform/swap/_temporary/0/task_local517689383_0004_m_000000
2015-02-12 09:42:31,756 INFO mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) - file:/Users/soberoi/src/compatibility/cascading.compatibility/horton/build/classes/test/data/apache.10.txt:0+1449
2015-02-12 09:42:31,756 INFO mapred.Task (Task.java:sendDone(1121)) - Task 'attempt_local517689383_0004_m_000000_0' done.
2015-02-12 09:42:31,756 INFO mapred.LocalJobRunner (LocalJobRunner.java:run(249)) - Finishing task: attempt_local517689383_0004_m_000000_0
2015-02-12 09:42:31,756 INFO mapred.LocalJobRunner (LocalJobRunner.java:runTasks(456)) - map task executor complete.
2015-02-12 09:42:31,770 INFO util.Hadoop18TapUtil (Hadoop18TapUtil.java:cleanTempPath(219)) - deleting temp path /Users/soberoi/src/compatibility/cascading.compatibility/horton/build/test/output/hadoop2-mr1/basicpipesplatform/swap/_temporary
2015-02-12 09:42:31,795 INFO mapred.FileInputFormat (FileInputFormat.java:listStatus(247)) - Total input paths to process : 1
2015-02-12 09:42:31,797 INFO cascading.PlatformTestCase (PlatformTestCase.java:tearDown(209)) - copying to local /Users/soberoi/src/compatibility/cascading.compatibility/horton/build/test/output//hadoop2-mr1/basicpipesplatform/swap
2015-02-12 09:42:31,828 INFO util.HadoopUtil (HadoopUtil.java:findMainClass(336)) - using default application jar, may cause class not found exceptions on the cluster
2015-02-12 09:42:31,829 INFO planner.HadoopPlanner (HadoopPlanner.java:initialize(225)) - using application jar: /Users/soberoi/.gradle/caches/modules-2/files-2.1/cascading/cascading-hadoop2-mr1/2.6.3/9ae016e2b8d246b9c1de9168e715d9f1b971aa4a/cascading-hadoop2-mr1-2.6.3.jar
2015-02-12 09:42:31,836 INFO Configuration.deprecation (Configuration.java:warnOnceIfDeprecated(1051)) - mapred.used.genericoptionsparser is deprecated. Instead, use mapreduce.client.genericoptionsparser.used
2015-02-12 09:42:31,840 INFO Configuration.deprecation (Configuration.java:warnOnceIfDeprecated(1051)) - mapred.job.tracker is deprecated. Instead, use mapreduce.jobtracker.address
2015-02-12 09:42:31,840 INFO Configuration.deprecation (Configuration.java:warnOnceIfDeprecated(1051)) - mapred.output.compress is deprecated. Instead, use mapreduce.output.fileoutputformat.compress
2015-02-12 09:42:31,848 INFO flow.Flow (BaseFlow.java:logInfo(1378)) - [test] starting
2015-02-12 09:42:31,849 INFO flow.Flow (BaseFlow.java:logInfo(1378)) - [test] source: Hfs["TextLine[[0:1]->[ALL]]"]["/Users/soberoi/src/compatibility/cascading.compatibility/horton/build/classes/test/data/lower+upper.txt"]
2015-02-12 09:42:31,849 INFO flow.Flow (BaseFlow.java:logInfo(1378)) - [test] sink: Hfs["TextLine[['offset', 'line']->[ALL]]"]["/Users/soberoi/src/compatibility/cascading.compatibility/horton/build/test/output/hadoop2-mr1/basicpipesplatform/ungrouped"]
2015-02-12 09:42:31,849 INFO flow.Flow (BaseFlow.java:logInfo(1378)) - [test] parallel execution is enabled: false
2015-02-12 09:42:31,849 INFO flow.Flow (BaseFlow.java:logInfo(1378)) - [test] starting jobs: 1
2015-02-12 09:42:31,849 INFO flow.Flow (BaseFlow.java:logInfo(1378)) - [test] allocating threads: 1
2015-02-12 09:42:31,850 INFO flow.FlowStep (BaseFlowStep.java:logInfo(834)) - [test] starting step: (1/1) ...icpipesplatform/ungrouped
2015-02-12 09:42:31,850 INFO jvm.JvmMetrics (JvmMetrics.java:init(71)) - Cannot initialize JVM Metrics with processName=JobTracker, sessionId= - already initialized
2015-02-12 09:42:31,852 INFO jvm.JvmMetrics (JvmMetrics.java:init(71)) - Cannot initialize JVM Metrics with processName=JobTracker, sessionId= - already initialized
2015-02-12 09:42:31,877 INFO mapred.FileInputFormat (FileInputFormat.java:listStatus(247)) - Total input paths to process : 1
2015-02-12 09:42:31,895 INFO mapreduce.JobSubmitter (JobSubmitter.java:submitJobInternal(494)) - number of splits:1
2015-02-12 09:42:31,908 INFO mapreduce.JobSubmitter (JobSubmitter.java:printTokens(583)) - Submitting tokens for job: job_local1865538857_0005
2015-02-12 09:42:31,957 INFO mapreduce.Job (Job.java:submit(1300)) - The url to track the job: http://localhost:8080/
2015-02-12 09:42:31,957 INFO mapred.LocalJobRunner (LocalJobRunner.java:createOutputCommitter(471)) - OutputCommitter set in config null
2015-02-12 09:42:31,957 INFO flow.FlowStep (BaseFlowStep.java:logInfo(834)) - [test] submitted hadoop job: job_local1865538857_0005
2015-02-12 09:42:31,957 INFO flow.FlowStep (BaseFlowStep.java:logInfo(834)) - [test] tracking url: http://localhost:8080/
2015-02-12 09:42:31,957 INFO mapred.LocalJobRunner (LocalJobRunner.java:createOutputCommitter(489)) - OutputCommitter is org.apache.hadoop.mapred.FileOutputCommitter
2015-02-12 09:42:31,959 INFO mapred.LocalJobRunner (LocalJobRunner.java:runTasks(448)) - Waiting for map tasks
2015-02-12 09:42:31,959 INFO mapred.LocalJobRunner (LocalJobRunner.java:run(224)) - Starting task: attempt_local1865538857_0005_m_000000_0
2015-02-12 09:42:31,961 INFO util.ProcfsBasedProcessTree (ProcfsBasedProcessTree.java:isAvailable(181)) - ProcfsBasedProcessTree currently is supported only on Linux.
2015-02-12 09:42:31,961 INFO mapred.Task (Task.java:initialize(587)) - Using ResourceCalculatorProcessTree : null
2015-02-12 09:42:31,962 INFO io.MultiInputSplit (MultiInputSplit.java:readFields(161)) - current split input path: file:/Users/soberoi/src/compatibility/cascading.compatibility/horton/build/classes/test/data/lower+upper.txt
2015-02-12 09:42:31,962 INFO mapred.MapTask (MapTask.java:updateJobWithSplit(482)) - Processing split: cascading.tap.hadoop.io.MultiInputSplit@4c46491b
2015-02-12 09:42:31,963 INFO mapred.MapTask (MapTask.java:runOldMapper(436)) - numReduceTasks: 0
2015-02-12 09:42:31,969 INFO hadoop.FlowMapper (FlowMapper.java:configure(71)) - cascading version: 2.6.3
2015-02-12 09:42:31,970 INFO hadoop.FlowMapper (FlowMapper.java:configure(72)) - child jvm opts: -Xmx200m
2015-02-12 09:42:31,976 INFO hadoop.FlowMapper (FlowMapper.java:configure(87)) - sourcing from: Hfs["TextLine[[0:1]->[ALL]]"]["/Users/soberoi/src/compatibility/cascading.compatibility/horton/build/classes/test/data/lower+upper.txt"]
2015-02-12 09:42:31,976 INFO hadoop.FlowMapper (FlowMapper.java:configure(90)) - sinking to: Hfs["TextLine[['offset', 'line']->[ALL]]"]["/Users/soberoi/src/compatibility/cascading.compatibility/horton/build/test/output/hadoop2-mr1/basicpipesplatform/ungrouped"]
2015-02-12 09:42:31,979 INFO mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) -
2015-02-12 09:42:31,979 INFO mapred.Task (Task.java:done(1001)) - Task:attempt_local1865538857_0005_m_000000_0 is done. And is in the process of committing
2015-02-12 09:42:31,980 INFO mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) -
2015-02-12 09:42:31,981 INFO mapred.Task (Task.java:commit(1162)) - Task attempt_local1865538857_0005_m_000000_0 is allowed to commit now
2015-02-12 09:42:31,982 INFO output.FileOutputCommitter (FileOutputCommitter.java:commitTask(439)) - Saved output of task 'attempt_local1865538857_0005_m_000000_0' to file:/Users/soberoi/src/compatibility/cascading.compatibility/horton/build/test/output/hadoop2-mr1/basicpipesplatform/ungrouped/_temporary/0/task_local1865538857_0005_m_000000
2015-02-12 09:42:31,983 INFO mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) - file:/Users/soberoi/src/compatibility/cascading.compatibility/horton/build/classes/test/data/lower+upper.txt:0+30
2015-02-12 09:42:31,983 INFO mapred.Task (Task.java:sendDone(1121)) - Task 'attempt_local1865538857_0005_m_000000_0' done.
2015-02-12 09:42:31,983 INFO mapred.LocalJobRunner (LocalJobRunner.java:run(249)) - Finishing task: attempt_local1865538857_0005_m_000000_0
2015-02-12 09:42:31,983 INFO mapred.LocalJobRunner (LocalJobRunner.java:runTasks(456)) - map task executor complete.
2015-02-12 09:42:32,006 INFO util.Hadoop18TapUtil (Hadoop18TapUtil.java:cleanTempPath(219)) - deleting temp path /Users/soberoi/src/compatibility/cascading.compatibility/horton/build/test/output/hadoop2-mr1/basicpipesplatform/ungrouped/_temporary
2015-02-12 09:42:32,029 INFO mapred.FileInputFormat (FileInputFormat.java:listStatus(247)) - Total input paths to process : 1
2015-02-12 09:42:32,031 INFO cascading.PlatformTestCase (PlatformTestCase.java:tearDown(209)) - copying to local /Users/soberoi/src/compatibility/cascading.compatibility/horton/build/test/output//hadoop2-mr1/basicpipesplatform/ungrouped
2015-02-12 09:42:32,043 INFO util.HadoopUtil (HadoopUtil.java:findMainClass(336)) - using default application jar, may cause class not found exceptions on the cluster
2015-02-12 09:42:32,044 INFO planner.HadoopPlanner (HadoopPlanner.java:initialize(225)) - using application jar: /Users/soberoi/.gradle/caches/modules-2/files-2.1/cascading/cascading-hadoop2-mr1/2.6.3/9ae016e2b8d246b9c1de9168e715d9f1b971aa4a/cascading-hadoop2-mr1-2.6.3.jar
2015-02-12 09:42:32,050 INFO Configuration.deprecation (Configuration.java:warnOnceIfDeprecated(1051)) - mapred.used.genericoptionsparser is deprecated. Instead, use mapreduce.client.genericoptionsparser.used
2015-02-12 09:42:32,051 INFO Configuration.deprecation (Configuration.java:warnOnceIfDeprecated(1051)) - mapred.job.tracker is deprecated. Instead, use mapreduce.jobtracker.address
2015-02-12 09:42:32,052 INFO Configuration.deprecation (Configuration.java:warnOnceIfDeprecated(1051)) - mapred.output.compress is deprecated. Instead, use mapreduce.output.fileoutputformat.compress
2015-02-12 09:42:32,057 INFO flow.Flow (BaseFlow.java:logInfo(1378)) - [test] starting
2015-02-12 09:42:32,057 INFO flow.Flow (BaseFlow.java:logInfo(1378)) - [test] source: Hfs["TextLine[[0:1]->[ALL]]"]["/Users/soberoi/src/compatibility/cascading.compatibility/horton/build/classes/test/data/apache.10.txt"]
2015-02-12 09:42:32,057 INFO flow.Flow (BaseFlow.java:logInfo(1378)) - [test] sink: Hfs["TextLine[['offset', 'line']->[ALL]]"]["/Users/soberoi/src/compatibility/cascading.compatibility/horton/build/test/output/hadoop2-mr1/basicpipesplatform/simplerelative"]
2015-02-12 09:42:32,057 INFO flow.Flow (BaseFlow.java:logInfo(1378)) - [test] parallel execution is enabled: false
2015-02-12 09:42:32,057 INFO flow.Flow (BaseFlow.java:logInfo(1378)) - [test] starting jobs: 1
2015-02-12 09:42:32,058 INFO flow.Flow (BaseFlow.java:logInfo(1378)) - [test] allocating threads: 1
2015-02-12 09:42:32,058 INFO flow.FlowStep (BaseFlowStep.java:logInfo(834)) - [test] starting step: (1/1) ...esplatform/simplerelative
2015-02-12 09:42:32,059 INFO jvm.JvmMetrics (JvmMetrics.java:init(71)) - Cannot initialize JVM Metrics with processName=JobTracker, sessionId= - already initialized
2015-02-12 09:42:32,060 INFO jvm.JvmMetrics (JvmMetrics.java:init(71)) - Cannot initialize JVM Metrics with processName=JobTracker, sessionId= - already initialized
2015-02-12 09:42:32,086 INFO mapred.FileInputFormat (FileInputFormat.java:listStatus(247)) - Total input paths to process : 1
2015-02-12 09:42:32,103 INFO mapreduce.JobSubmitter (JobSubmitter.java:submitJobInternal(494)) - number of splits:1
2015-02-12 09:42:32,117 INFO mapreduce.JobSubmitter (JobSubmitter.java:printTokens(583)) - Submitting tokens for job: job_local153526861_0006
2015-02-12 09:42:32,163 INFO mapreduce.Job (Job.java:submit(1300)) - The url to track the job: http://localhost:8080/
2015-02-12 09:42:32,164 INFO mapred.LocalJobRunner (LocalJobRunner.java:createOutputCommitter(471)) - OutputCommitter set in config null
2015-02-12 09:42:32,164 INFO flow.FlowStep (BaseFlowStep.java:logInfo(834)) - [test] submitted hadoop job: job_local153526861_0006
2015-02-12 09:42:32,164 INFO mapred.LocalJobRunner (LocalJobRunner.java:createOutputCommitter(489)) - OutputCommitter is org.apache.hadoop.mapred.FileOutputCommitter
2015-02-12 09:42:32,164 INFO flow.FlowStep (BaseFlowStep.java:logInfo(834)) - [test] tracking url: http://localhost:8080/
2015-02-12 09:42:32,166 INFO mapred.LocalJobRunner (LocalJobRunner.java:runTasks(448)) - Waiting for map tasks
2015-02-12 09:42:32,166 INFO mapred.LocalJobRunner (LocalJobRunner.java:run(224)) - Starting task: attempt_local153526861_0006_m_000000_0
2015-02-12 09:42:32,168 INFO util.ProcfsBasedProcessTree (ProcfsBasedProcessTree.java:isAvailable(181)) - ProcfsBasedProcessTree currently is supported only on Linux.
2015-02-12 09:42:32,168 INFO mapred.Task (Task.java:initialize(587)) - Using ResourceCalculatorProcessTree : null
2015-02-12 09:42:32,169 INFO io.MultiInputSplit (MultiInputSplit.java:readFields(161)) - current split input path: file:/Users/soberoi/src/compatibility/cascading.compatibility/horton/build/classes/test/data/apache.10.txt
2015-02-12 09:42:32,169 INFO mapred.MapTask (MapTask.java:updateJobWithSplit(482)) - Processing split: cascading.tap.hadoop.io.MultiInputSplit@1bcac9ea
2015-02-12 09:42:32,170 INFO mapred.MapTask (MapTask.java:runOldMapper(436)) - numReduceTasks: 1
2015-02-12 09:42:32,220 INFO mapred.MapTask (MapTask.java:setEquator(1202)) - (EQUATOR) 0 kvi 26214396(104857584)
2015-02-12 09:42:32,220 INFO mapred.MapTask (MapTask.java:init(995)) - mapreduce.task.io.sort.mb: 100
2015-02-12 09:42:32,220 INFO mapred.MapTask (MapTask.java:init(996)) - soft limit at 83886080
2015-02-12 09:42:32,220 INFO mapred.MapTask (MapTask.java:init(997)) - bufstart = 0; bufvoid = 104857600
2015-02-12 09:42:32,220 INFO mapred.MapTask (MapTask.java:init(998)) - kvstart = 26214396; length = 6553600
2015-02-12 09:42:32,221 INFO mapred.MapTask (MapTask.java:createSortingCollector(402)) - Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer
2015-02-12 09:42:32,221 INFO hadoop.FlowMapper (FlowMapper.java:configure(71)) - cascading version: 2.6.3
2015-02-12 09:42:32,221 INFO hadoop.FlowMapper (FlowMapper.java:configure(72)) - child jvm opts: -Xmx200m
2015-02-12 09:42:32,227 INFO hadoop.FlowMapper (FlowMapper.java:configure(87)) - sourcing from: Hfs["TextLine[[0:1]->[ALL]]"]["/Users/soberoi/src/compatibility/cascading.compatibility/horton/build/classes/test/data/apache.10.txt"]
2015-02-12 09:42:32,227 INFO hadoop.FlowMapper (FlowMapper.java:configure(90)) - sinking to: GroupBy(test)[by:[{1}:0]]
2015-02-12 09:42:32,229 INFO mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) -
2015-02-12 09:42:32,229 INFO mapred.MapTask (MapTask.java:flush(1457)) - Starting flush of map output
2015-02-12 09:42:32,229 INFO mapred.MapTask (MapTask.java:flush(1475)) - Spilling map output
2015-02-12 09:42:32,229 INFO mapred.MapTask (MapTask.java:flush(1476)) - bufstart = 0; bufend = 210; bufvoid = 104857600
2015-02-12 09:42:32,229 INFO mapred.MapTask (MapTask.java:flush(1478)) - kvstart = 26214396(104857584); kvend = 26214360(104857440); length = 37/6553600
2015-02-12 09:42:32,231 INFO mapred.MapTask (MapTask.java:sortAndSpill(1660)) - Finished spill 0
2015-02-12 09:42:32,232 INFO mapred.Task (Task.java:done(1001)) - Task:attempt_local153526861_0006_m_000000_0 is done. And is in the process of committing
2015-02-12 09:42:32,234 INFO mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) - file:/Users/soberoi/src/compatibility/cascading.compatibility/horton/build/classes/test/data/apache.10.txt:0+1449
2015-02-12 09:42:32,234 INFO mapred.Task (Task.java:sendDone(1121)) - Task 'attempt_local153526861_0006_m_000000_0' done.
2015-02-12 09:42:32,234 INFO mapred.LocalJobRunner (LocalJobRunner.java:run(249)) - Finishing task: attempt_local153526861_0006_m_000000_0
2015-02-12 09:42:32,234 INFO mapred.LocalJobRunner (LocalJobRunner.java:runTasks(456)) - map task executor complete.
2015-02-12 09:42:32,235 INFO mapred.LocalJobRunner (LocalJobRunner.java:runTasks(448)) - Waiting for reduce tasks
2015-02-12 09:42:32,235 INFO mapred.LocalJobRunner (LocalJobRunner.java:run(302)) - Starting task: attempt_local153526861_0006_r_000000_0
2015-02-12 09:42:32,236 INFO util.ProcfsBasedProcessTree (ProcfsBasedProcessTree.java:isAvailable(181)) - ProcfsBasedProcessTree currently is supported only on Linux.
2015-02-12 09:42:32,237 INFO mapred.Task (Task.java:initialize(587)) - Using ResourceCalculatorProcessTree : null
2015-02-12 09:42:32,237 INFO mapred.ReduceTask (ReduceTask.java:run(362)) - Using ShuffleConsumerPlugin: org.apache.hadoop.mapreduce.task.reduce.Shuffle@68d10be4
2015-02-12 09:42:32,237 INFO reduce.MergeManagerImpl (MergeManagerImpl.java:<init>(196)) - MergerManager: memoryLimit=493250144, maxSingleShuffleLimit=123312536, mergeThreshold=325545120, ioSortFactor=10, memToMemMergeOutputsThreshold=10
2015-02-12 09:42:32,238 INFO reduce.EventFetcher (EventFetcher.java:run(61)) - attempt_local153526861_0006_r_000000_0 Thread started: EventFetcher for fetching Map Completion Events
2015-02-12 09:42:32,239 INFO reduce.LocalFetcher (LocalFetcher.java:copyMapOutput(141)) - localfetcher#4 about to shuffle output of map attempt_local153526861_0006_m_000000_0 decomp: 232 len: 236 to MEMORY
2015-02-12 09:42:32,239 INFO reduce.InMemoryMapOutput (InMemoryMapOutput.java:shuffle(100)) - Read 232 bytes from map-output for attempt_local153526861_0006_m_000000_0
2015-02-12 09:42:32,240 INFO reduce.MergeManagerImpl (MergeManagerImpl.java:closeInMemoryFile(314)) - closeInMemoryFile -> map-output of size: 232, inMemoryMapOutputs.size() -> 1, commitMemory -> 0, usedMemory ->232
2015-02-12 09:42:32,240 INFO reduce.EventFetcher (EventFetcher.java:run(76)) - EventFetcher is interrupted.. Returning
2015-02-12 09:42:32,241 INFO mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) - 1 / 1 copied.
2015-02-12 09:42:32,241 INFO reduce.MergeManagerImpl (MergeManagerImpl.java:finalMerge(674)) - finalMerge called with 1 in-memory map-outputs and 0 on-disk map-outputs
2015-02-12 09:42:32,242 INFO mapred.Merger (Merger.java:merge(597)) - Merging 1 sorted segments
2015-02-12 09:42:32,242 INFO mapred.Merger (Merger.java:merge(696)) - Down to the last merge-pass, with 1 segments left of total size: 211 bytes
2015-02-12 09:42:32,243 INFO reduce.MergeManagerImpl (MergeManagerImpl.java:finalMerge(751)) - Merged 1 segments, 232 bytes to disk to satisfy reduce memory limit
2015-02-12 09:42:32,244 INFO reduce.MergeManagerImpl (MergeManagerImpl.java:finalMerge(781)) - Merging 1 files, 236 bytes from disk
2015-02-12 09:42:32,244 INFO reduce.MergeManagerImpl (MergeManagerImpl.java:finalMerge(796)) - Merging 0 segments, 0 bytes from memory into reduce
2015-02-12 09:42:32,244 INFO mapred.Merger (Merger.java:merge(597)) - Merging 1 sorted segments
2015-02-12 09:42:32,244 INFO mapred.Merger (Merger.java:merge(696)) - Down to the last merge-pass, with 1 segments left of total size: 211 bytes
2015-02-12 09:42:32,245 INFO mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) - 1 / 1 copied.
2015-02-12 09:42:32,245 INFO hadoop.FlowReducer (FlowReducer.java:configure(78)) - cascading version: 2.6.3
2015-02-12 09:42:32,245 INFO hadoop.FlowReducer (FlowReducer.java:configure(79)) - child jvm opts: -Xmx200m
2015-02-12 09:42:32,250 INFO hadoop.FlowReducer (FlowReducer.java:configure(97)) - sourcing from: GroupBy(test)[by:[{1}:0]]
2015-02-12 09:42:32,250 INFO hadoop.FlowReducer (FlowReducer.java:configure(100)) - sinking to: Hfs["TextLine[['offset', 'line']->[ALL]]"]["/Users/soberoi/src/compatibility/cascading.compatibility/horton/build/test/output/hadoop2-mr1/basicpipesplatform/simplerelative"]
2015-02-12 09:42:32,259 INFO mapred.Task (Task.java:done(1001)) - Task:attempt_local153526861_0006_r_000000_0 is done. And is in the process of committing
2015-02-12 09:42:32,260 INFO mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) - 1 / 1 copied.
2015-02-12 09:42:32,260 INFO mapred.Task (Task.java:commit(1162)) - Task attempt_local153526861_0006_r_000000_0 is allowed to commit now
2015-02-12 09:42:32,261 INFO output.FileOutputCommitter (FileOutputCommitter.java:commitTask(439)) - Saved output of task 'attempt_local153526861_0006_r_000000_0' to file:/Users/soberoi/src/compatibility/cascading.compatibility/horton/build/test/output/hadoop2-mr1/basicpipesplatform/simplerelative/_temporary/0/task_local153526861_0006_r_000000
2015-02-12 09:42:32,261 INFO mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) - reduce > reduce
2015-02-12 09:42:32,262 INFO mapred.Task (Task.java:sendDone(1121)) - Task 'attempt_local153526861_0006_r_000000_0' done.
2015-02-12 09:42:32,262 INFO mapred.LocalJobRunner (LocalJobRunner.java:run(325)) - Finishing task: attempt_local153526861_0006_r_000000_0
2015-02-12 09:42:32,262 INFO mapred.LocalJobRunner (LocalJobRunner.java:runTasks(456)) - reduce task executor complete.
2015-02-12 09:42:32,286 INFO util.Hadoop18TapUtil (Hadoop18TapUtil.java:cleanTempPath(219)) - deleting temp path /Users/soberoi/src/compatibility/cascading.compatibility/horton/build/test/output/hadoop2-mr1/basicpipesplatform/simplerelative/_temporary
2015-02-12 09:42:32,312 INFO mapred.FileInputFormat (FileInputFormat.java:listStatus(247)) - Total input paths to process : 1
2015-02-12 09:42:32,313 INFO cascading.PlatformTestCase (PlatformTestCase.java:tearDown(209)) - copying to local /Users/soberoi/src/compatibility/cascading.compatibility/horton/build/test/output//hadoop2-mr1/basicpipesplatform/simplerelative
2015-02-12 09:42:32,391 INFO util.HadoopUtil (HadoopUtil.java:findMainClass(336)) - using default application jar, may cause class not found exceptions on the cluster
2015-02-12 09:42:32,392 INFO planner.HadoopPlanner (HadoopPlanner.java:initialize(225)) - using application jar: /Users/soberoi/.gradle/caches/modules-2/files-2.1/cascading/cascading-hadoop2-mr1/2.6.3/9ae016e2b8d246b9c1de9168e715d9f1b971aa4a/cascading-hadoop2-mr1-2.6.3.jar
2015-02-12 09:42:32,399 INFO Configuration.deprecation (Configuration.java:warnOnceIfDeprecated(1051)) - mapred.used.genericoptionsparser is deprecated. Instead, use mapreduce.client.genericoptionsparser.used
2015-02-12 09:42:32,401 INFO Configuration.deprecation (Configuration.java:warnOnceIfDeprecated(1051)) - mapred.job.tracker is deprecated. Instead, use mapreduce.jobtracker.address
2015-02-12 09:42:32,401 INFO Configuration.deprecation (Configuration.java:warnOnceIfDeprecated(1051)) - mapred.output.compress is deprecated. Instead, use mapreduce.output.fileoutputformat.compress
2015-02-12 09:42:32,411 INFO flow.Flow (BaseFlow.java:logInfo(1378)) - [test] starting
2015-02-12 09:42:32,411 INFO flow.Flow (BaseFlow.java:logInfo(1378)) - [test] source: Hfs["TextLine[[0:1]->[ALL]]"]["/Users/soberoi/src/compatibility/cascading.compatibility/horton/build/classes/test/data/lower.txt"]
2015-02-12 09:42:32,411 INFO flow.Flow (BaseFlow.java:logInfo(1378)) - [test] sink: Hfs["TextLine[[0]->[ALL]]"]["/Users/soberoi/src/compatibility/cascading.compatibility/horton/build/test/output/hadoop2-mr1/basicpipesplatform/simpleresult"]
2015-02-12 09:42:32,412 INFO flow.Flow (BaseFlow.java:logInfo(1378)) - [test] parallel execution is enabled: false
2015-02-12 09:42:32,412 INFO flow.Flow (BaseFlow.java:logInfo(1378)) - [test] starting jobs: 1
2015-02-12 09:42:32,412 INFO flow.Flow (BaseFlow.java:logInfo(1378)) - [test] allocating threads: 1
2015-02-12 09:42:32,412 INFO flow.FlowStep (BaseFlowStep.java:logInfo(834)) - [test] starting step: (1/1) ...ipesplatform/simpleresult
2015-02-12 09:42:32,413 INFO jvm.JvmMetrics (JvmMetrics.java:init(71)) - Cannot initialize JVM Metrics with processName=JobTracker, sessionId= - already initialized
2015-02-12 09:42:32,414 INFO jvm.JvmMetrics (JvmMetrics.java:init(71)) - Cannot initialize JVM Metrics with processName=JobTracker, sessionId= - already initialized
2015-02-12 09:42:32,435 INFO mapred.FileInputFormat (FileInputFormat.java:listStatus(247)) - Total input paths to process : 1
2015-02-12 09:42:32,453 INFO mapreduce.JobSubmitter (JobSubmitter.java:submitJobInternal(494)) - number of splits:1
2015-02-12 09:42:32,485 INFO mapreduce.JobSubmitter (JobSubmitter.java:printTokens(583)) - Submitting tokens for job: job_local1892918645_0007
2015-02-12 09:42:32,534 INFO mapreduce.Job (Job.java:submit(1300)) - The url to track the job: http://localhost:8080/
2015-02-12 09:42:32,534 INFO mapred.LocalJobRunner (LocalJobRunner.java:createOutputCommitter(471)) - OutputCommitter set in config null
2015-02-12 09:42:32,534 INFO flow.FlowStep (BaseFlowStep.java:logInfo(834)) - [test] submitted hadoop job: job_local1892918645_0007
2015-02-12 09:42:32,534 INFO mapred.LocalJobRunner (LocalJobRunner.java:createOutputCommitter(489)) - OutputCommitter is org.apache.hadoop.mapred.FileOutputCommitter
2015-02-12 09:42:32,535 INFO flow.FlowStep (BaseFlowStep.java:logInfo(834)) - [test] tracking url: http://localhost:8080/
2015-02-12 09:42:32,537 INFO mapred.LocalJobRunner (LocalJobRunner.java:runTasks(448)) - Waiting for map tasks
2015-02-12 09:42:32,537 INFO mapred.LocalJobRunner (LocalJobRunner.java:run(224)) - Starting task: attempt_local1892918645_0007_m_000000_0
2015-02-12 09:42:32,538 INFO util.ProcfsBasedProcessTree (ProcfsBasedProcessTree.java:isAvailable(181)) - ProcfsBasedProcessTree currently is supported only on Linux.
2015-02-12 09:42:32,538 INFO mapred.Task (Task.java:initialize(587)) - Using ResourceCalculatorProcessTree : null
2015-02-12 09:42:32,539 INFO io.MultiInputSplit (MultiInputSplit.java:readFields(161)) - current split input path: file:/Users/soberoi/src/compatibility/cascading.compatibility/horton/build/classes/test/data/lower.txt
2015-02-12 09:42:32,539 INFO mapred.MapTask (MapTask.java:updateJobWithSplit(482)) - Processing split: cascading.tap.hadoop.io.MultiInputSplit@5c4e4be3
2015-02-12 09:42:32,540 INFO mapred.MapTask (MapTask.java:runOldMapper(436)) - numReduceTasks: 1
2015-02-12 09:42:32,552 INFO mapred.MapTask (MapTask.java:setEquator(1202)) - (EQUATOR) 0 kvi 26214396(104857584)
2015-02-12 09:42:32,552 INFO mapred.MapTask (MapTask.java:init(995)) - mapreduce.task.io.sort.mb: 100
2015-02-12 09:42:32,552 INFO mapred.MapTask (MapTask.java:init(996)) - soft limit at 83886080
2015-02-12 09:42:32,552 INFO mapred.MapTask (MapTask.java:init(997)) - bufstart = 0; bufvoid = 104857600
2015-02-12 09:42:32,552 INFO mapred.MapTask (MapTask.java:init(998)) - kvstart = 26214396; length = 6553600
2015-02-12 09:42:32,553 INFO mapred.MapTask (MapTask.java:createSortingCollector(402)) - Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer
2015-02-12 09:42:32,553 INFO hadoop.FlowMapper (FlowMapper.java:configure(71)) - cascading version: 2.6.3
2015-02-12 09:42:32,553 INFO hadoop.FlowMapper (FlowMapper.java:configure(72)) - child jvm opts: -Xmx200m
2015-02-12 09:42:32,560 INFO hadoop.FlowMapper (FlowMapper.java:configure(87)) - sourcing from: Hfs["TextLine[[0:1]->[ALL]]"]["/Users/soberoi/src/compatibility/cascading.compatibility/horton/build/classes/test/data/lower.txt"]
2015-02-12 09:42:32,560 INFO hadoop.FlowMapper (FlowMapper.java:configure(90)) - sinking to: GroupBy(test)[by:[{1}:0]]
2015-02-12 09:42:32,614 INFO mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) -
2015-02-12 09:42:32,614 INFO mapred.MapTask (MapTask.java:flush(1457)) - Starting flush of map output
2015-02-12 09:42:32,615 INFO mapred.MapTask (MapTask.java:flush(1475)) - Spilling map output
2015-02-12 09:42:32,615 INFO mapred.MapTask (MapTask.java:flush(1476)) - bufstart = 0; bufend = 22; bufvoid = 104857600
2015-02-12 09:42:32,615 INFO mapred.MapTask (MapTask.java:flush(1478)) - kvstart = 26214396(104857584); kvend = 26214392(104857568); length = 5/6553600
2015-02-12 09:42:32,616 INFO mapred.MapTask (MapTask.java:sortAndSpill(1660)) - Finished spill 0
2015-02-12 09:42:32,618 INFO mapred.Task (Task.java:done(1001)) - Task:attempt_local1892918645_0007_m_000000_0 is done. And is in the process of committing
2015-02-12 09:42:32,619 INFO mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) - file:/Users/soberoi/src/compatibility/cascading.compatibility/horton/build/classes/test/data/lower.txt:0+19
2015-02-12 09:42:32,619 INFO mapred.Task (Task.java:sendDone(1121)) - Task 'attempt_local1892918645_0007_m_000000_0' done.
2015-02-12 09:42:32,619 INFO mapred.LocalJobRunner (LocalJobRunner.java:run(249)) - Finishing task: attempt_local1892918645_0007_m_000000_0
2015-02-12 09:42:32,619 INFO mapred.LocalJobRunner (LocalJobRunner.java:runTasks(456)) - map task executor complete.
2015-02-12 09:42:32,620 INFO mapred.LocalJobRunner (LocalJobRunner.java:runTasks(448)) - Waiting for reduce tasks
2015-02-12 09:42:32,620 INFO mapred.LocalJobRunner (LocalJobRunner.java:run(302)) - Starting task: attempt_local1892918645_0007_r_000000_0
2015-02-12 09:42:32,622 INFO util.ProcfsBasedProcessTree (ProcfsBasedProcessTree.java:isAvailable(181)) - ProcfsBasedProcessTree currently is supported only on Linux.
2015-02-12 09:42:32,622 INFO mapred.Task (Task.java:initialize(587)) - Using ResourceCalculatorProcessTree : null
2015-02-12 09:42:32,622 INFO mapred.ReduceTask (ReduceTask.java:run(362)) - Using ShuffleConsumerPlugin: org.apache.hadoop.mapreduce.task.reduce.Shuffle@45125002
2015-02-12 09:42:32,622 INFO reduce.MergeManagerImpl (MergeManagerImpl.java:<init>(196)) - MergerManager: memoryLimit=493250144, maxSingleShuffleLimit=123312536, mergeThreshold=325545120, ioSortFactor=10, memToMemMergeOutputsThreshold=10
2015-02-12 09:42:32,623 INFO reduce.EventFetcher (EventFetcher.java:run(61)) - attempt_local1892918645_0007_r_000000_0 Thread started: EventFetcher for fetching Map Completion Events
2015-02-12 09:42:32,624 INFO reduce.LocalFetcher (LocalFetcher.java:copyMapOutput(141)) - localfetcher#5 about to shuffle output of map attempt_local1892918645_0007_m_000000_0 decomp: 28 len: 32 to MEMORY
2015-02-12 09:42:32,624 INFO reduce.InMemoryMapOutput (InMemoryMapOutput.java:shuffle(100)) - Read 28 bytes from map-output for attempt_local1892918645_0007_m_000000_0
2015-02-12 09:42:32,624 INFO reduce.MergeManagerImpl (MergeManagerImpl.java:closeInMemoryFile(314)) - closeInMemoryFile -> map-output of size: 28, inMemoryMapOutputs.size() -> 1, commitMemory -> 0, usedMemory ->28
2015-02-12 09:42:32,625 INFO reduce.EventFetcher (EventFetcher.java:run(76)) - EventFetcher is interrupted.. Returning
2015-02-12 09:42:32,625 INFO mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) - 1 / 1 copied.
2015-02-12 09:42:32,626 INFO reduce.MergeManagerImpl (MergeManagerImpl.java:finalMerge(674)) - finalMerge called with 1 in-memory map-outputs and 0 on-disk map-outputs
2015-02-12 09:42:32,627 INFO mapred.Merger (Merger.java:merge(597)) - Merging 1 sorted segments
2015-02-12 09:42:32,627 INFO mapred.Merger (Merger.java:merge(696)) - Down to the last merge-pass, with 1 segments left of total size: 17 bytes
2015-02-12 09:42:32,628 INFO reduce.MergeManagerImpl (MergeManagerImpl.java:finalMerge(751)) - Merged 1 segments, 28 bytes to disk to satisfy reduce memory limit
2015-02-12 09:42:32,629 INFO reduce.MergeManagerImpl (MergeManagerImpl.java:finalMerge(781)) - Merging 1 files, 32 bytes from disk
2015-02-12 09:42:32,629 INFO reduce.MergeManagerImpl (MergeManagerImpl.java:finalMerge(796)) - Merging 0 segments, 0 bytes from memory into reduce
2015-02-12 09:42:32,629 INFO mapred.Merger (Merger.java:merge(597)) - Merging 1 sorted segments
2015-02-12 09:42:32,629 INFO mapred.Merger (Merger.java:merge(696)) - Down to the last merge-pass, with 1 segments left of total size: 17 bytes
2015-02-12 09:42:32,630 INFO mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) - 1 / 1 copied.
2015-02-12 09:42:32,630 INFO hadoop.FlowReducer (FlowReducer.java:configure(78)) - cascading version: 2.6.3
2015-02-12 09:42:32,630 INFO hadoop.FlowReducer (FlowReducer.java:configure(79)) - child jvm opts: -Xmx200m
2015-02-12 09:42:32,638 INFO hadoop.FlowReducer (FlowReducer.java:configure(97)) - sourcing from: GroupBy(test)[by:[{1}:0]]
2015-02-12 09:42:32,638 INFO hadoop.FlowReducer (FlowReducer.java:configure(100)) - sinking to: Hfs["TextLine[[0]->[ALL]]"]["/Users/soberoi/src/compatibility/cascading.compatibility/horton/build/test/output/hadoop2-mr1/basicpipesplatform/simpleresult"]
2015-02-12 09:42:32,648 INFO mapred.Task (Task.java:done(1001)) - Task:attempt_local1892918645_0007_r_000000_0 is done. And is in the process of committing
2015-02-12 09:42:32,650 INFO mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) - 1 / 1 copied.
2015-02-12 09:42:32,650 INFO mapred.Task (Task.java:commit(1162)) - Task attempt_local1892918645_0007_r_000000_0 is allowed to commit now
2015-02-12 09:42:32,651 INFO output.FileOutputCommitter (FileOutputCommitter.java:commitTask(439)) - Saved output of task 'attempt_local1892918645_0007_r_000000_0' to file:/Users/soberoi/src/compatibility/cascading.compatibility/horton/build/test/output/hadoop2-mr1/basicpipesplatform/simpleresult/_temporary/0/task_local1892918645_0007_r_000000
2015-02-12 09:42:32,652 INFO mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) - reduce > reduce
2015-02-12 09:42:32,652 INFO mapred.Task (Task.java:sendDone(1121)) - Task 'attempt_local1892918645_0007_r_000000_0' done.
2015-02-12 09:42:32,652 INFO mapred.LocalJobRunner (LocalJobRunner.java:run(325)) - Finishing task: attempt_local1892918645_0007_r_000000_0
2015-02-12 09:42:32,653 INFO mapred.LocalJobRunner (LocalJobRunner.java:runTasks(456)) - reduce task executor complete.
2015-02-12 09:42:32,676 INFO util.Hadoop18TapUtil (Hadoop18TapUtil.java:cleanTempPath(219)) - deleting temp path /Users/soberoi/src/compatibility/cascading.compatibility/horton/build/test/output/hadoop2-mr1/basicpipesplatform/simpleresult/_temporary
2015-02-12 09:42:32,711 INFO mapred.FileInputFormat (FileInputFormat.java:listStatus(247)) - Total input paths to process : 1
2015-02-12 09:42:32,713 INFO cascading.PlatformTestCase (PlatformTestCase.java:tearDown(209)) - copying to local /Users/soberoi/src/compatibility/cascading.compatibility/horton/build/test/output//hadoop2-mr1/basicpipesplatform/simpleresult
2015-02-12 09:42:32,736 INFO util.HadoopUtil (HadoopUtil.java:findMainClass(336)) - using default application jar, may cause class not found exceptions on the cluster
2015-02-12 09:42:32,736 INFO planner.HadoopPlanner (HadoopPlanner.java:initialize(225)) - using application jar: /Users/soberoi/.gradle/caches/modules-2/files-2.1/cascading/cascading-hadoop2-mr1/2.6.3/9ae016e2b8d246b9c1de9168e715d9f1b971aa4a/cascading-hadoop2-mr1-2.6.3.jar
2015-02-12 09:42:32,744 INFO Configuration.deprecation (Configuration.java:warnOnceIfDeprecated(1051)) - mapred.used.genericoptionsparser is deprecated. Instead, use mapreduce.client.genericoptionsparser.used
2015-02-12 09:42:32,747 INFO Configuration.deprecation (Configuration.java:warnOnceIfDeprecated(1051)) - mapred.job.tracker is deprecated. Instead, use mapreduce.jobtracker.address
2015-02-12 09:42:32,747 INFO Configuration.deprecation (Configuration.java:warnOnceIfDeprecated(1051)) - mapred.output.compress is deprecated. Instead, use mapreduce.output.fileoutputformat.compress
2015-02-12 09:42:32,753 INFO flow.Flow (BaseFlow.java:logInfo(1378)) - [test] starting
2015-02-12 09:42:32,754 INFO flow.Flow (BaseFlow.java:logInfo(1378)) - [test] source: Hfs["TextLine[[0:1]->[ALL]]"]["/Users/soberoi/src/compatibility/cascading.compatibility/horton/build/classes/test/data/apache.10.txt"]
2015-02-12 09:42:32,754 INFO flow.Flow (BaseFlow.java:logInfo(1378)) - [test] sink: Hfs["TextLine[['offset', 'line']->[ALL]]"]["/Users/soberoi/src/compatibility/cascading.compatibility/horton/build/test/output/hadoop2-mr1/basicpipesplatform/filterall"]
2015-02-12 09:42:32,754 INFO flow.Flow (BaseFlow.java:logInfo(1378)) - [test] parallel execution is enabled: false
2015-02-12 09:42:32,754 INFO flow.Flow (BaseFlow.java:logInfo(1378)) - [test] starting jobs: 1
2015-02-12 09:42:32,754 INFO flow.Flow (BaseFlow.java:logInfo(1378)) - [test] allocating threads: 1
2015-02-12 09:42:32,755 INFO flow.FlowStep (BaseFlowStep.java:logInfo(834)) - [test] starting step: (1/1) ...icpipesplatform/filterall
2015-02-12 09:42:32,756 INFO jvm.JvmMetrics (JvmMetrics.java:init(71)) - Cannot initialize JVM Metrics with processName=JobTracker, sessionId= - already initialized
2015-02-12 09:42:32,757 INFO jvm.JvmMetrics (JvmMetrics.java:init(71)) - Cannot initialize JVM Metrics with processName=JobTracker, sessionId= - already initialized
2015-02-12 09:42:32,787 INFO mapred.FileInputFormat (FileInputFormat.java:listStatus(247)) - Total input paths to process : 1
2015-02-12 09:42:32,811 INFO mapreduce.JobSubmitter (JobSubmitter.java:submitJobInternal(494)) - number of splits:1
2015-02-12 09:42:32,833 INFO mapreduce.JobSubmitter (JobSubmitter.java:printTokens(583)) - Submitting tokens for job: job_local1877223176_0008
2015-02-12 09:42:32,916 INFO mapreduce.Job (Job.java:submit(1300)) - The url to track the job: http://localhost:8080/
2015-02-12 09:42:32,916 INFO mapred.LocalJobRunner (LocalJobRunner.java:createOutputCommitter(471)) - OutputCommitter set in config null
2015-02-12 09:42:32,917 INFO mapred.LocalJobRunner (LocalJobRunner.java:createOutputCommitter(489)) - OutputCommitter is org.apache.hadoop.mapred.FileOutputCommitter
2015-02-12 09:42:32,916 INFO flow.FlowStep (BaseFlowStep.java:logInfo(834)) - [test] submitted hadoop job: job_local1877223176_0008
2015-02-12 09:42:32,918 INFO flow.FlowStep (BaseFlowStep.java:logInfo(834)) - [test] tracking url: http://localhost:8080/
2015-02-12 09:42:32,919 INFO mapred.LocalJobRunner (LocalJobRunner.java:runTasks(448)) - Waiting for map tasks
2015-02-12 09:42:32,919 INFO mapred.LocalJobRunner (LocalJobRunner.java:run(224)) - Starting task: attempt_local1877223176_0008_m_000000_0
2015-02-12 09:42:32,921 INFO util.ProcfsBasedProcessTree (ProcfsBasedProcessTree.java:isAvailable(181)) - ProcfsBasedProcessTree currently is supported only on Linux.
2015-02-12 09:42:32,921 INFO mapred.Task (Task.java:initialize(587)) - Using ResourceCalculatorProcessTree : null
2015-02-12 09:42:32,922 INFO io.MultiInputSplit (MultiInputSplit.java:readFields(161)) - current split input path: file:/Users/soberoi/src/compatibility/cascading.compatibility/horton/build/classes/test/data/apache.10.txt
2015-02-12 09:42:32,923 INFO mapred.MapTask (MapTask.java:updateJobWithSplit(482)) - Processing split: cascading.tap.hadoop.io.MultiInputSplit@6e5814ac
2015-02-12 09:42:32,924 INFO mapred.MapTask (MapTask.java:runOldMapper(436)) - numReduceTasks: 0
2015-02-12 09:42:32,932 INFO hadoop.FlowMapper (FlowMapper.java:configure(71)) - cascading version: 2.6.3
2015-02-12 09:42:32,932 INFO hadoop.FlowMapper (FlowMapper.java:configure(72)) - child jvm opts: -Xmx200m
2015-02-12 09:42:32,938 INFO hadoop.FlowMapper (FlowMapper.java:configure(87)) - sourcing from: Hfs["TextLine[[0:1]->[ALL]]"]["/Users/soberoi/src/compatibility/cascading.compatibility/horton/build/classes/test/data/apache.10.txt"]
2015-02-12 09:42:32,938 INFO hadoop.FlowMapper (FlowMapper.java:configure(90)) - sinking to: Hfs["TextLine[['offset', 'line']->[ALL]]"]["/Users/soberoi/src/compatibility/cascading.compatibility/horton/build/test/output/hadoop2-mr1/basicpipesplatform/filterall"]
2015-02-12 09:42:32,941 INFO mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) -
2015-02-12 09:42:32,941 INFO mapred.Task (Task.java:done(1001)) - Task:attempt_local1877223176_0008_m_000000_0 is done. And is in the process of committing
2015-02-12 09:42:32,942 INFO mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) -
2015-02-12 09:42:32,943 INFO mapred.Task (Task.java:commit(1162)) - Task attempt_local1877223176_0008_m_000000_0 is allowed to commit now
2015-02-12 09:42:32,943 INFO output.FileOutputCommitter (FileOutputCommitter.java:commitTask(439)) - Saved output of task 'attempt_local1877223176_0008_m_000000_0' to file:/Users/soberoi/src/compatibility/cascading.compatibility/horton/build/test/output/hadoop2-mr1/basicpipesplatform/filterall/_temporary/0/task_local1877223176_0008_m_000000
2015-02-12 09:42:32,945 INFO mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) - file:/Users/soberoi/src/compatibility/cascading.compatibility/horton/build/classes/test/data/apache.10.txt:0+1449
2015-02-12 09:42:32,945 INFO mapred.Task (Task.java:sendDone(1121)) - Task 'attempt_local1877223176_0008_m_000000_0' done.
2015-02-12 09:42:32,945 INFO mapred.LocalJobRunner (LocalJobRunner.java:run(249)) - Finishing task: attempt_local1877223176_0008_m_000000_0
2015-02-12 09:42:32,945 INFO mapred.LocalJobRunner (LocalJobRunner.java:runTasks(456)) - map task executor complete.
2015-02-12 09:42:32,966 INFO util.Hadoop18TapUtil (Hadoop18TapUtil.java:cleanTempPath(219)) - deleting temp path /Users/soberoi/src/compatibility/cascading.compatibility/horton/build/test/output/hadoop2-mr1/basicpipesplatform/filterall/_temporary
2015-02-12 09:42:33,004 INFO mapred.FileInputFormat (FileInputFormat.java:listStatus(247)) - Total input paths to process : 1
2015-02-12 09:42:33,006 INFO cascading.PlatformTestCase (PlatformTestCase.java:tearDown(209)) - copying to local /Users/soberoi/src/compatibility/cascading.compatibility/horton/build/test/output//hadoop2-mr1/basicpipesplatform/filterall
2015-02-12 09:42:33,032 INFO util.HadoopUtil (HadoopUtil.java:findMainClass(336)) - using default application jar, may cause class not found exceptions on the cluster
2015-02-12 09:42:33,032 INFO planner.HadoopPlanner (HadoopPlanner.java:initialize(225)) - using application jar: /Users/soberoi/.gradle/caches/modules-2/files-2.1/cascading/cascading-hadoop2-mr1/2.6.3/9ae016e2b8d246b9c1de9168e715d9f1b971aa4a/cascading-hadoop2-mr1-2.6.3.jar
2015-02-12 09:42:33,046 INFO Configuration.deprecation (Configuration.java:warnOnceIfDeprecated(1051)) - mapred.used.genericoptionsparser is deprecated. Instead, use mapreduce.client.genericoptionsparser.used
2015-02-12 09:42:33,051 INFO Configuration.deprecation (Configuration.java:warnOnceIfDeprecated(1051)) - mapred.job.tracker is deprecated. Instead, use mapreduce.jobtracker.address
2015-02-12 09:42:33,052 INFO Configuration.deprecation (Configuration.java:warnOnceIfDeprecated(1051)) - mapred.output.compress is deprecated. Instead, use mapreduce.output.fileoutputformat.compress
2015-02-12 09:42:33,063 INFO flow.Flow (BaseFlow.java:logInfo(1378)) - [lower*upper] starting
2015-02-12 09:42:33,063 INFO flow.Flow (BaseFlow.java:logInfo(1378)) - [lower*upper] source: Hfs["TextLine[[0:1]->[ALL]]"]["/Users/soberoi/src/compatibility/cascading.compatibility/horton/build/classes/test/data/upper.txt"]
2015-02-12 09:42:33,063 INFO flow.Flow (BaseFlow.java:logInfo(1378)) - [lower*upper] source: Hfs["TextLine[[0:1]->[ALL]]"]["/Users/soberoi/src/compatibility/cascading.compatibility/horton/build/classes/test/data/lower.txt"]
2015-02-12 09:42:33,063 INFO flow.Flow (BaseFlow.java:logInfo(1378)) - [lower*upper] sink: Hfs["TextLine[[0]->[ALL]]"]["/Users/soberoi/src/compatibility/cascading.compatibility/horton/build/test/output/hadoop2-mr1/basicpipesplatform/complexcogroup"]
2015-02-12 09:42:33,064 INFO flow.Flow (BaseFlow.java:logInfo(1378)) - [lower*upper] parallel execution is enabled: false
2015-02-12 09:42:33,064 INFO flow.Flow (BaseFlow.java:logInfo(1378)) - [lower*upper] starting jobs: 1
2015-02-12 09:42:33,064 INFO flow.Flow (BaseFlow.java:logInfo(1378)) - [lower*upper] allocating threads: 1
2015-02-12 09:42:33,064 INFO flow.FlowStep (BaseFlowStep.java:logInfo(834)) - [lower*upper] starting step: (1/1) ...esplatform/complexcogroup
2015-02-12 09:42:33,065 INFO jvm.JvmMetrics (JvmMetrics.java:init(71)) - Cannot initialize JVM Metrics with processName=JobTracker, sessionId= - already initialized
2015-02-12 09:42:33,066 INFO jvm.JvmMetrics (JvmMetrics.java:init(71)) - Cannot initialize JVM Metrics with processName=JobTracker, sessionId= - already initialized
2015-02-12 09:42:33,098 INFO mapred.FileInputFormat (FileInputFormat.java:listStatus(247)) - Total input paths to process : 1
2015-02-12 09:42:33,100 INFO mapred.FileInputFormat (FileInputFormat.java:listStatus(247)) - Total input paths to process : 1
2015-02-12 09:42:33,142 INFO mapreduce.JobSubmitter (JobSubmitter.java:submitJobInternal(494)) - number of splits:2
2015-02-12 09:42:33,161 INFO mapreduce.JobSubmitter (JobSubmitter.java:printTokens(583)) - Submitting tokens for job: job_local1738896954_0009
2015-02-12 09:42:33,237 INFO mapreduce.Job (Job.java:submit(1300)) - The url to track the job: http://localhost:8080/
2015-02-12 09:42:33,238 INFO mapred.LocalJobRunner (LocalJobRunner.java:createOutputCommitter(471)) - OutputCommitter set in config null
2015-02-12 09:42:33,238 INFO mapred.LocalJobRunner (LocalJobRunner.java:createOutputCommitter(489)) - OutputCommitter is org.apache.hadoop.mapred.FileOutputCommitter
2015-02-12 09:42:33,238 INFO flow.FlowStep (BaseFlowStep.java:logInfo(834)) - [lower*upper] submitted hadoop job: job_local1738896954_0009
2015-02-12 09:42:33,238 INFO flow.FlowStep (BaseFlowStep.java:logInfo(834)) - [lower*upper] tracking url: http://localhost:8080/
2015-02-12 09:42:33,241 INFO mapred.LocalJobRunner (LocalJobRunner.java:runTasks(448)) - Waiting for map tasks
2015-02-12 09:42:33,241 INFO mapred.LocalJobRunner (LocalJobRunner.java:run(224)) - Starting task: attempt_local1738896954_0009_m_000000_0
2015-02-12 09:42:33,242 INFO util.ProcfsBasedProcessTree (ProcfsBasedProcessTree.java:isAvailable(181)) - ProcfsBasedProcessTree currently is supported only on Linux.
2015-02-12 09:42:33,242 INFO mapred.Task (Task.java:initialize(587)) - Using ResourceCalculatorProcessTree : null
2015-02-12 09:42:33,243 INFO io.MultiInputSplit (MultiInputSplit.java:readFields(161)) - current split input path: file:/Users/soberoi/src/compatibility/cascading.compatibility/horton/build/classes/test/data/upper.txt
2015-02-12 09:42:33,244 INFO mapred.MapTask (MapTask.java:updateJobWithSplit(482)) - Processing split: cascading.tap.hadoop.io.MultiInputSplit@154b37e5
2015-02-12 09:42:33,245 INFO mapred.MapTask (MapTask.java:runOldMapper(436)) - numReduceTasks: 1
2015-02-12 09:42:33,258 INFO mapred.MapTask (MapTask.java:setEquator(1202)) - (EQUATOR) 0 kvi 26214396(104857584)
2015-02-12 09:42:33,259 INFO mapred.MapTask (MapTask.java:init(995)) - mapreduce.task.io.sort.mb: 100
2015-02-12 09:42:33,259 INFO mapred.MapTask (MapTask.java:init(996)) - soft limit at 83886080
2015-02-12 09:42:33,259 INFO mapred.MapTask (MapTask.java:init(997)) - bufstart = 0; bufvoid = 104857600
2015-02-12 09:42:33,259 INFO mapred.MapTask (MapTask.java:init(998)) - kvstart = 26214396; length = 6553600
2015-02-12 09:42:33,261 INFO mapred.MapTask (MapTask.java:createSortingCollector(402)) - Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer
2015-02-12 09:42:33,261 INFO hadoop.FlowMapper (FlowMapper.java:configure(71)) - cascading version: 2.6.3
2015-02-12 09:42:33,261 INFO hadoop.FlowMapper (FlowMapper.java:configure(72)) - child jvm opts: -Xmx200m
2015-02-12 09:42:33,268 INFO hadoop.FlowMapper (FlowMapper.java:configure(87)) - sourcing from: Hfs["TextLine[[0:1]->[ALL]]"]["/Users/soberoi/src/compatibility/cascading.compatibility/horton/build/classes/test/data/upper.txt"]
2015-02-12 09:42:33,268 INFO hadoop.FlowMapper (FlowMapper.java:configure(90)) - sinking to: CoGroup(lower*upper)[by: lower:[{1}:0] upper:[{1}:0]]
2015-02-12 09:42:33,271 INFO mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) -
2015-02-12 09:42:33,271 INFO mapred.MapTask (MapTask.java:flush(1457)) - Starting flush of map output
2015-02-12 09:42:33,271 INFO mapred.MapTask (MapTask.java:flush(1475)) - Spilling map output
2015-02-12 09:42:33,271 INFO mapred.MapTask (MapTask.java:flush(1476)) - bufstart = 0; bufend = 85; bufvoid = 104857600
2015-02-12 09:42:33,272 INFO mapred.MapTask (MapTask.java:flush(1478)) - kvstart = 26214396(104857584); kvend = 26214380(104857520); length = 17/6553600
2015-02-12 09:42:33,274 INFO mapred.MapTask (MapTask.java:sortAndSpill(1660)) - Finished spill 0
2015-02-12 09:42:33,276 INFO mapred.Task (Task.java:done(1001)) - Task:attempt_local1738896954_0009_m_000000_0 is done. And is in the process of committing
2015-02-12 09:42:33,277 INFO mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) - file:/Users/soberoi/src/compatibility/cascading.compatibility/horton/build/classes/test/data/upper.txt:0+19
2015-02-12 09:42:33,278 INFO mapred.Task (Task.java:sendDone(1121)) - Task 'attempt_local1738896954_0009_m_000000_0' done.
2015-02-12 09:42:33,278 INFO mapred.LocalJobRunner (LocalJobRunner.java:run(249)) - Finishing task: attempt_local1738896954_0009_m_000000_0
2015-02-12 09:42:33,278 INFO mapred.LocalJobRunner (LocalJobRunner.java:run(224)) - Starting task: attempt_local1738896954_0009_m_000001_0
2015-02-12 09:42:33,279 INFO util.ProcfsBasedProcessTree (ProcfsBasedProcessTree.java:isAvailable(181)) - ProcfsBasedProcessTree currently is supported only on Linux.
2015-02-12 09:42:33,279 INFO mapred.Task (Task.java:initialize(587)) - Using ResourceCalculatorProcessTree : null
2015-02-12 09:42:33,281 INFO io.MultiInputSplit (MultiInputSplit.java:readFields(161)) - current split input path: file:/Users/soberoi/src/compatibility/cascading.compatibility/horton/build/classes/test/data/lower.txt
2015-02-12 09:42:33,281 INFO mapred.MapTask (MapTask.java:updateJobWithSplit(482)) - Processing split: cascading.tap.hadoop.io.MultiInputSplit@2fd6e265
2015-02-12 09:42:33,282 INFO mapred.MapTask (MapTask.java:runOldMapper(436)) - numReduceTasks: 1
2015-02-12 09:42:33,308 INFO mapred.MapTask (MapTask.java:setEquator(1202)) - (EQUATOR) 0 kvi 26214396(104857584)
2015-02-12 09:42:33,309 INFO mapred.MapTask (MapTask.java:init(995)) - mapreduce.task.io.sort.mb: 100
2015-02-12 09:42:33,309 INFO mapred.MapTask (MapTask.java:init(996)) - soft limit at 83886080
2015-02-12 09:42:33,309 INFO mapred.MapTask (MapTask.java:init(997)) - bufstart = 0; bufvoid = 104857600
2015-02-12 09:42:33,309 INFO mapred.MapTask (MapTask.java:init(998)) - kvstart = 26214396; length = 6553600
2015-02-12 09:42:33,310 INFO mapred.MapTask (MapTask.java:createSortingCollector(402)) - Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer
2015-02-12 09:42:33,310 INFO hadoop.FlowMapper (FlowMapper.java:configure(71)) - cascading version: 2.6.3
2015-02-12 09:42:33,310 INFO hadoop.FlowMapper (FlowMapper.java:configure(72)) - child jvm opts: -Xmx200m
2015-02-12 09:42:33,316 INFO hadoop.FlowMapper (FlowMapper.java:configure(87)) - sourcing from: Hfs["TextLine[[0:1]->[ALL]]"]["/Users/soberoi/src/compatibility/cascading.compatibility/horton/build/classes/test/data/lower.txt"]
2015-02-12 09:42:33,317 INFO hadoop.FlowMapper (FlowMapper.java:configure(90)) - sinking to: CoGroup(lower*upper)[by: lower:[{1}:0] upper:[{1}:0]]
2015-02-12 09:42:33,319 INFO mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) -
2015-02-12 09:42:33,319 INFO mapred.MapTask (MapTask.java:flush(1457)) - Starting flush of map output
2015-02-12 09:42:33,319 INFO mapred.MapTask (MapTask.java:flush(1475)) - Spilling map output
2015-02-12 09:42:33,319 INFO mapred.MapTask (MapTask.java:flush(1476)) - bufstart = 0; bufend = 85; bufvoid = 104857600
2015-02-12 09:42:33,320 INFO mapred.MapTask (MapTask.java:flush(1478)) - kvstart = 26214396(104857584); kvend = 26214380(104857520); length = 17/6553600
2015-02-12 09:42:33,322 INFO mapred.MapTask (MapTask.java:sortAndSpill(1660)) - Finished spill 0
2015-02-12 09:42:33,324 INFO mapred.Task (Task.java:done(1001)) - Task:attempt_local1738896954_0009_m_000001_0 is done. And is in the process of committing
2015-02-12 09:42:33,326 INFO mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) - file:/Users/soberoi/src/compatibility/cascading.compatibility/horton/build/classes/test/data/lower.txt:0+19
2015-02-12 09:42:33,326 INFO mapred.Task (Task.java:sendDone(1121)) - Task 'attempt_local1738896954_0009_m_000001_0' done.
2015-02-12 09:42:33,326 INFO mapred.LocalJobRunner (LocalJobRunner.java:run(249)) - Finishing task: attempt_local1738896954_0009_m_000001_0
2015-02-12 09:42:33,326 INFO mapred.LocalJobRunner (LocalJobRunner.java:runTasks(456)) - map task executor complete.
2015-02-12 09:42:33,327 INFO mapred.LocalJobRunner (LocalJobRunner.java:runTasks(448)) - Waiting for reduce tasks
2015-02-12 09:42:33,327 INFO mapred.LocalJobRunner (LocalJobRunner.java:run(302)) - Starting task: attempt_local1738896954_0009_r_000000_0
2015-02-12 09:42:33,329 INFO util.ProcfsBasedProcessTree (ProcfsBasedProcessTree.java:isAvailable(181)) - ProcfsBasedProcessTree currently is supported only on Linux.
2015-02-12 09:42:33,329 INFO mapred.Task (Task.java:initialize(587)) - Using ResourceCalculatorProcessTree : null
2015-02-12 09:42:33,329 INFO mapred.ReduceTask (ReduceTask.java:run(362)) - Using ShuffleConsumerPlugin: org.apache.hadoop.mapreduce.task.reduce.Shuffle@400a2506
2015-02-12 09:42:33,330 INFO reduce.MergeManagerImpl (MergeManagerImpl.java:<init>(196)) - MergerManager: memoryLimit=493250144, maxSingleShuffleLimit=123312536, mergeThreshold=325545120, ioSortFactor=10, memToMemMergeOutputsThreshold=10
2015-02-12 09:42:33,344 INFO reduce.EventFetcher (EventFetcher.java:run(61)) - attempt_local1738896954_0009_r_000000_0 Thread started: EventFetcher for fetching Map Completion Events
2015-02-12 09:42:33,346 INFO reduce.LocalFetcher (LocalFetcher.java:copyMapOutput(141)) - localfetcher#6 about to shuffle output of map attempt_local1738896954_0009_m_000000_0 decomp: 97 len: 101 to MEMORY
2015-02-12 09:42:33,346 INFO reduce.InMemoryMapOutput (InMemoryMapOutput.java:shuffle(100)) - Read 97 bytes from map-output for attempt_local1738896954_0009_m_000000_0
2015-02-12 09:42:33,346 INFO reduce.MergeManagerImpl (MergeManagerImpl.java:closeInMemoryFile(314)) - closeInMemoryFile -> map-output of size: 97, inMemoryMapOutputs.size() -> 1, commitMemory -> 0, usedMemory ->97
2015-02-12 09:42:33,348 INFO reduce.LocalFetcher (LocalFetcher.java:copyMapOutput(141)) - localfetcher#6 about to shuffle output of map attempt_local1738896954_0009_m_000001_0 decomp: 97 len: 101 to MEMORY
2015-02-12 09:42:33,348 INFO reduce.InMemoryMapOutput (InMemoryMapOutput.java:shuffle(100)) - Read 97 bytes from map-output for attempt_local1738896954_0009_m_000001_0
2015-02-12 09:42:33,348 INFO reduce.MergeManagerImpl (MergeManagerImpl.java:closeInMemoryFile(314)) - closeInMemoryFile -> map-output of size: 97, inMemoryMapOutputs.size() -> 2, commitMemory -> 97, usedMemory ->194
2015-02-12 09:42:33,349 INFO reduce.EventFetcher (EventFetcher.java:run(76)) - EventFetcher is interrupted.. Returning
2015-02-12 09:42:33,350 INFO mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) - 2 / 2 copied.
2015-02-12 09:42:33,350 INFO reduce.MergeManagerImpl (MergeManagerImpl.java:finalMerge(674)) - finalMerge called with 2 in-memory map-outputs and 0 on-disk map-outputs
2015-02-12 09:42:33,352 INFO mapred.Merger (Merger.java:merge(597)) - Merging 2 sorted segments
2015-02-12 09:42:33,352 INFO mapred.Merger (Merger.java:merge(696)) - Down to the last merge-pass, with 2 segments left of total size: 174 bytes
2015-02-12 09:42:33,353 INFO reduce.MergeManagerImpl (MergeManagerImpl.java:finalMerge(751)) - Merged 2 segments, 194 bytes to disk to satisfy reduce memory limit
2015-02-12 09:42:33,354 INFO reduce.MergeManagerImpl (MergeManagerImpl.java:finalMerge(781)) - Merging 1 files, 196 bytes from disk
2015-02-12 09:42:33,354 INFO reduce.MergeManagerImpl (MergeManagerImpl.java:finalMerge(796)) - Merging 0 segments, 0 bytes from memory into reduce
2015-02-12 09:42:33,354 INFO mapred.Merger (Merger.java:merge(597)) - Merging 1 sorted segments
2015-02-12 09:42:33,354 INFO mapred.Merger (Merger.java:merge(696)) - Down to the last merge-pass, with 1 segments left of total size: 182 bytes
2015-02-12 09:42:33,355 INFO mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) - 2 / 2 copied.
2015-02-12 09:42:33,355 INFO hadoop.FlowReducer (FlowReducer.java:configure(78)) - cascading version: 2.6.3
2015-02-12 09:42:33,355 INFO hadoop.FlowReducer (FlowReducer.java:configure(79)) - child jvm opts: -Xmx200m
2015-02-12 09:42:33,360 INFO hadoop.FlowReducer (FlowReducer.java:configure(97)) - sourcing from: CoGroup(lower*upper)[by: lower:[{1}:0] upper:[{1}:0]]
2015-02-12 09:42:33,360 INFO hadoop.FlowReducer (FlowReducer.java:configure(100)) - sinking to: Hfs["TextLine[[0]->[ALL]]"]["/Users/soberoi/src/compatibility/cascading.compatibility/horton/build/test/output/hadoop2-mr1/basicpipesplatform/complexcogroup"]
2015-02-12 09:42:33,374 INFO collect.SpillableTupleList (SpillableTupleList.java:getCodecClass(105)) - attempting to load codec: org.apache.hadoop.io.compress.GzipCodec
2015-02-12 09:42:33,374 INFO collect.SpillableTupleList (SpillableTupleList.java:getCodecClass(110)) - found codec: org.apache.hadoop.io.compress.GzipCodec
2015-02-12 09:42:33,381 INFO mapred.Task (Task.java:done(1001)) - Task:attempt_local1738896954_0009_r_000000_0 is done. And is in the process of committing
2015-02-12 09:42:33,382 INFO mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) - 2 / 2 copied.
2015-02-12 09:42:33,382 INFO mapred.Task (Task.java:commit(1162)) - Task attempt_local1738896954_0009_r_000000_0 is allowed to commit now
2015-02-12 09:42:33,383 INFO output.FileOutputCommitter (FileOutputCommitter.java:commitTask(439)) - Saved output of task 'attempt_local1738896954_0009_r_000000_0' to file:/Users/soberoi/src/compatibility/cascading.compatibility/horton/build/test/output/hadoop2-mr1/basicpipesplatform/complexcogroup/_temporary/0/task_local1738896954_0009_r_000000
2015-02-12 09:42:33,384 INFO mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) - reduce > reduce
2015-02-12 09:42:33,384 INFO mapred.Task (Task.java:sendDone(1121)) - Task 'attempt_local1738896954_0009_r_000000_0' done.
2015-02-12 09:42:33,385 INFO mapred.LocalJobRunner (LocalJobRunner.java:run(325)) - Finishing task: attempt_local1738896954_0009_r_000000_0
2015-02-12 09:42:33,385 INFO mapred.LocalJobRunner (LocalJobRunner.java:runTasks(456)) - reduce task executor complete.
2015-02-12 09:42:33,405 INFO util.Hadoop18TapUtil (Hadoop18TapUtil.java:cleanTempPath(219)) - deleting temp path /Users/soberoi/src/compatibility/cascading.compatibility/horton/build/test/output/hadoop2-mr1/basicpipesplatform/complexcogroup/_temporary
2015-02-12 09:42:33,439 INFO mapred.FileInputFormat (FileInputFormat.java:listStatus(247)) - Total input paths to process : 1
2015-02-12 09:42:33,476 INFO mapred.FileInputFormat (FileInputFormat.java:listStatus(247)) - Total input paths to process : 1
2015-02-12 09:42:33,478 INFO cascading.PlatformTestCase (PlatformTestCase.java:tearDown(209)) - copying to local /Users/soberoi/src/compatibility/cascading.compatibility/horton/build/test/output//hadoop2-mr1/basicpipesplatform/complexcogroup
2015-02-12 09:42:33,499 INFO util.HadoopUtil (HadoopUtil.java:findMainClass(336)) - using default application jar, may cause class not found exceptions on the cluster
2015-02-12 09:42:33,500 INFO planner.HadoopPlanner (HadoopPlanner.java:initialize(225)) - using application jar: /Users/soberoi/.gradle/caches/modules-2/files-2.1/cascading/cascading-hadoop2-mr1/2.6.3/9ae016e2b8d246b9c1de9168e715d9f1b971aa4a/cascading-hadoop2-mr1-2.6.3.jar
2015-02-12 09:42:33,506 INFO Configuration.deprecation (Configuration.java:warnOnceIfDeprecated(1051)) - mapred.used.genericoptionsparser is deprecated. Instead, use mapreduce.client.genericoptionsparser.used
2015-02-12 09:42:33,508 INFO Configuration.deprecation (Configuration.java:warnOnceIfDeprecated(1051)) - mapred.job.tracker is deprecated. Instead, use mapreduce.jobtracker.address
2015-02-12 09:42:33,509 INFO Configuration.deprecation (Configuration.java:warnOnceIfDeprecated(1051)) - mapred.output.compress is deprecated. Instead, use mapreduce.output.fileoutputformat.compress
2015-02-12 09:42:33,514 INFO flow.Flow (BaseFlow.java:logInfo(1378)) - [test] starting
2015-02-12 09:42:33,515 INFO flow.Flow (BaseFlow.java:logInfo(1378)) - [test] source: Hfs["TextLine[[0:1]->[ALL]]"]["/Users/soberoi/src/compatibility/cascading.compatibility/horton/build/classes/test/data/apache.10.txt"]
2015-02-12 09:42:33,515 INFO flow.Flow (BaseFlow.java:logInfo(1378)) - [test] sink: Hfs["TextLine[['offset', 'line']->[ALL]]"]["/Users/soberoi/src/compatibility/cascading.compatibility/horton/build/test/output/hadoop2-mr1/basicpipesplatform/replace"]
2015-02-12 09:42:33,515 INFO flow.Flow (BaseFlow.java:logInfo(1378)) - [test] parallel execution is enabled: false
2015-02-12 09:42:33,515 INFO flow.Flow (BaseFlow.java:logInfo(1378)) - [test] starting jobs: 1
2015-02-12 09:42:33,515 INFO flow.Flow (BaseFlow.java:logInfo(1378)) - [test] allocating threads: 1
2015-02-12 09:42:33,515 INFO flow.FlowStep (BaseFlowStep.java:logInfo(834)) - [test] starting step: (1/1) ...asicpipesplatform/replace
2015-02-12 09:42:33,516 INFO jvm.JvmMetrics (JvmMetrics.java:init(71)) - Cannot initialize JVM Metrics with processName=JobTracker, sessionId= - already initialized
2015-02-12 09:42:33,517 INFO jvm.JvmMetrics (JvmMetrics.java:init(71)) - Cannot initialize JVM Metrics with processName=JobTracker, sessionId= - already initialized
2015-02-12 09:42:33,545 INFO mapred.FileInputFormat (FileInputFormat.java:listStatus(247)) - Total input paths to process : 1
2015-02-12 09:42:33,569 INFO mapreduce.JobSubmitter (JobSubmitter.java:submitJobInternal(494)) - number of splits:1
2015-02-12 09:42:33,587 INFO mapreduce.JobSubmitter (JobSubmitter.java:printTokens(583)) - Submitting tokens for job: job_local824987974_0010
2015-02-12 09:42:33,662 INFO mapreduce.Job (Job.java:submit(1300)) - The url to track the job: http://localhost:8080/
2015-02-12 09:42:33,663 INFO mapred.LocalJobRunner (LocalJobRunner.java:createOutputCommitter(471)) - OutputCommitter set in config null
2015-02-12 09:42:33,663 INFO mapred.LocalJobRunner (LocalJobRunner.java:createOutputCommitter(489)) - OutputCommitter is org.apache.hadoop.mapred.FileOutputCommitter
2015-02-12 09:42:33,663 INFO flow.FlowStep (BaseFlowStep.java:logInfo(834)) - [test] submitted hadoop job: job_local824987974_0010
2015-02-12 09:42:33,665 INFO flow.FlowStep (BaseFlowStep.java:logInfo(834)) - [test] tracking url: http://localhost:8080/
2015-02-12 09:42:33,666 INFO mapred.LocalJobRunner (LocalJobRunner.java:runTasks(448)) - Waiting for map tasks
2015-02-12 09:42:33,666 INFO mapred.LocalJobRunner (LocalJobRunner.java:run(224)) - Starting task: attempt_local824987974_0010_m_000000_0
2015-02-12 09:42:33,667 INFO util.ProcfsBasedProcessTree (ProcfsBasedProcessTree.java:isAvailable(181)) - ProcfsBasedProcessTree currently is supported only on Linux.
2015-02-12 09:42:33,667 INFO mapred.Task (Task.java:initialize(587)) - Using ResourceCalculatorProcessTree : null
2015-02-12 09:42:33,669 INFO io.MultiInputSplit (MultiInputSplit.java:readFields(161)) - current split input path: file:/Users/soberoi/src/compatibility/cascading.compatibility/horton/build/classes/test/data/apache.10.txt
2015-02-12 09:42:33,669 INFO mapred.MapTask (MapTask.java:updateJobWithSplit(482)) - Processing split: cascading.tap.hadoop.io.MultiInputSplit@779c8497
2015-02-12 09:42:33,670 INFO mapred.MapTask (MapTask.java:runOldMapper(436)) - numReduceTasks: 0
2015-02-12 09:42:33,679 INFO hadoop.FlowMapper (FlowMapper.java:configure(71)) - cascading version: 2.6.3
2015-02-12 09:42:33,679 INFO hadoop.FlowMapper (FlowMapper.java:configure(72)) - child jvm opts: -Xmx200m
2015-02-12 09:42:33,683 INFO hadoop.FlowMapper (FlowMapper.java:configure(87)) - sourcing from: Hfs["TextLine[[0:1]->[ALL]]"]["/Users/soberoi/src/compatibility/cascading.compatibility/horton/build/classes/test/data/apache.10.txt"]
2015-02-12 09:42:33,684 INFO hadoop.FlowMapper (FlowMapper.java:configure(90)) - sinking to: Hfs["TextLine[['offset', 'line']->[ALL]]"]["/Users/soberoi/src/compatibility/cascading.compatibility/horton/build/test/output/hadoop2-mr1/basicpipesplatform/replace"]
2015-02-12 09:42:33,686 INFO mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) -
2015-02-12 09:42:33,687 INFO mapred.Task (Task.java:done(1001)) - Task:attempt_local824987974_0010_m_000000_0 is done. And is in the process of committing
2015-02-12 09:42:33,688 INFO mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) -
2015-02-12 09:42:33,688 INFO mapred.Task (Task.java:commit(1162)) - Task attempt_local824987974_0010_m_000000_0 is allowed to commit now
2015-02-12 09:42:33,689 INFO output.FileOutputCommitter (FileOutputCommitter.java:commitTask(439)) - Saved output of task 'attempt_local824987974_0010_m_000000_0' to file:/Users/soberoi/src/compatibility/cascading.compatibility/horton/build/test/output/hadoop2-mr1/basicpipesplatform/replace/_temporary/0/task_local824987974_0010_m_000000
2015-02-12 09:42:33,690 INFO mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) - file:/Users/soberoi/src/compatibility/cascading.compatibility/horton/build/classes/test/data/apache.10.txt:0+1449
2015-02-12 09:42:33,690 INFO mapred.Task (Task.java:sendDone(1121)) - Task 'attempt_local824987974_0010_m_000000_0' done.
2015-02-12 09:42:33,690 INFO mapred.LocalJobRunner (LocalJobRunner.java:run(249)) - Finishing task: attempt_local824987974_0010_m_000000_0
2015-02-12 09:42:33,690 INFO mapred.LocalJobRunner (LocalJobRunner.java:runTasks(456)) - map task executor complete.
2015-02-12 09:42:33,702 INFO util.Hadoop18TapUtil (Hadoop18TapUtil.java:cleanTempPath(219)) - deleting temp path /Users/soberoi/src/compatibility/cascading.compatibility/horton/build/test/output/hadoop2-mr1/basicpipesplatform/replace/_temporary
2015-02-12 09:42:33,733 INFO mapred.FileInputFormat (FileInputFormat.java:listStatus(247)) - Total input paths to process : 1
2015-02-12 09:42:33,735 INFO cascading.PlatformTestCase (PlatformTestCase.java:tearDown(209)) - copying to local /Users/soberoi/src/compatibility/cascading.compatibility/horton/build/test/output//hadoop2-mr1/basicpipesplatform/replace
2015-02-12 09:42:33,748 INFO util.HadoopUtil (HadoopUtil.java:findMainClass(336)) - using default application jar, may cause class not found exceptions on the cluster
2015-02-12 09:42:33,749 INFO planner.HadoopPlanner (HadoopPlanner.java:initialize(225)) - using application jar: /Users/soberoi/.gradle/caches/modules-2/files-2.1/cascading/cascading-hadoop2-mr1/2.6.3/9ae016e2b8d246b9c1de9168e715d9f1b971aa4a/cascading-hadoop2-mr1-2.6.3.jar
2015-02-12 09:42:33,756 INFO Configuration.deprecation (Configuration.java:warnOnceIfDeprecated(1051)) - mapred.used.genericoptionsparser is deprecated. Instead, use mapreduce.client.genericoptionsparser.used
2015-02-12 09:42:33,759 INFO Configuration.deprecation (Configuration.java:warnOnceIfDeprecated(1051)) - mapred.job.tracker is deprecated. Instead, use mapreduce.jobtracker.address
2015-02-12 09:42:33,760 INFO Configuration.deprecation (Configuration.java:warnOnceIfDeprecated(1051)) - mapred.output.compress is deprecated. Instead, use mapreduce.output.fileoutputformat.compress
2015-02-12 09:42:33,766 INFO flow.Flow (BaseFlow.java:logInfo(1378)) - [count] starting
2015-02-12 09:42:33,766 INFO flow.Flow (BaseFlow.java:logInfo(1378)) - [count] source: Hfs["TextLine[[0:1]->[ALL]]"]["/Users/soberoi/src/compatibility/cascading.compatibility/horton/build/classes/test/data/ips.20.txt"]
2015-02-12 09:42:33,766 INFO flow.Flow (BaseFlow.java:logInfo(1378)) - [count] sink: Hfs["TextLine[[0]->[ALL]]"]["/Users/soberoi/src/compatibility/cascading.compatibility/horton/build/test/output/hadoop2-mr1/basicpipesplatform/count2"]
2015-02-12 09:42:33,767 INFO flow.Flow (BaseFlow.java:logInfo(1378)) - [count] parallel execution is enabled: false
2015-02-12 09:42:33,767 INFO flow.Flow (BaseFlow.java:logInfo(1378)) - [count] starting jobs: 1
2015-02-12 09:42:33,767 INFO flow.Flow (BaseFlow.java:logInfo(1378)) - [count] allocating threads: 1
2015-02-12 09:42:33,767 INFO flow.FlowStep (BaseFlowStep.java:logInfo(834)) - [count] starting step: (1/1) ...basicpipesplatform/count2
2015-02-12 09:42:33,768 INFO jvm.JvmMetrics (JvmMetrics.java:init(71)) - Cannot initialize JVM Metrics with processName=JobTracker, sessionId= - already initialized
2015-02-12 09:42:33,769 INFO jvm.JvmMetrics (JvmMetrics.java:init(71)) - Cannot initialize JVM Metrics with processName=JobTracker, sessionId= - already initialized
2015-02-12 09:42:33,795 INFO mapred.FileInputFormat (FileInputFormat.java:listStatus(247)) - Total input paths to process : 1
2015-02-12 09:42:33,820 INFO mapreduce.JobSubmitter (JobSubmitter.java:submitJobInternal(494)) - number of splits:1
2015-02-12 09:42:33,839 INFO mapreduce.JobSubmitter (JobSubmitter.java:printTokens(583)) - Submitting tokens for job: job_local1796527190_0011
2015-02-12 09:42:33,911 INFO mapreduce.Job (Job.java:submit(1300)) - The url to track the job: http://localhost:8080/
2015-02-12 09:42:33,912 INFO mapred.LocalJobRunner (LocalJobRunner.java:createOutputCommitter(471)) - OutputCommitter set in config null
2015-02-12 09:42:33,912 INFO mapred.LocalJobRunner (LocalJobRunner.java:createOutputCommitter(489)) - OutputCommitter is org.apache.hadoop.mapred.FileOutputCommitter
2015-02-12 09:42:33,912 INFO flow.FlowStep (BaseFlowStep.java:logInfo(834)) - [count] submitted hadoop job: job_local1796527190_0011
2015-02-12 09:42:33,912 INFO flow.FlowStep (BaseFlowStep.java:logInfo(834)) - [count] tracking url: http://localhost:8080/
2015-02-12 09:42:33,914 INFO mapred.LocalJobRunner (LocalJobRunner.java:runTasks(448)) - Waiting for map tasks
2015-02-12 09:42:33,914 INFO mapred.LocalJobRunner (LocalJobRunner.java:run(224)) - Starting task: attempt_local1796527190_0011_m_000000_0
2015-02-12 09:42:33,916 INFO util.ProcfsBasedProcessTree (ProcfsBasedProcessTree.java:isAvailable(181)) - ProcfsBasedProcessTree currently is supported only on Linux.
2015-02-12 09:42:33,916 INFO mapred.Task (Task.java:initialize(587)) - Using ResourceCalculatorProcessTree : null
2015-02-12 09:42:33,917 INFO io.MultiInputSplit (MultiInputSplit.java:readFields(161)) - current split input path: file:/Users/soberoi/src/compatibility/cascading.compatibility/horton/build/classes/test/data/ips.20.txt
2015-02-12 09:42:33,917 INFO mapred.MapTask (MapTask.java:updateJobWithSplit(482)) - Processing split: cascading.tap.hadoop.io.MultiInputSplit@67b74e36
2015-02-12 09:42:33,918 INFO mapred.MapTask (MapTask.java:runOldMapper(436)) - numReduceTasks: 1
2015-02-12 09:42:33,947 INFO mapred.MapTask (MapTask.java:setEquator(1202)) - (EQUATOR) 0 kvi 26214396(104857584)
2015-02-12 09:42:33,948 INFO mapred.MapTask (MapTask.java:init(995)) - mapreduce.task.io.sort.mb: 100
2015-02-12 09:42:33,949 INFO mapred.MapTask (MapTask.java:init(996)) - soft limit at 83886080
2015-02-12 09:42:33,949 INFO mapred.MapTask (MapTask.java:init(997)) - bufstart = 0; bufvoid = 104857600
2015-02-12 09:42:33,949 INFO mapred.MapTask (MapTask.java:init(998)) - kvstart = 26214396; length = 6553600
2015-02-12 09:42:33,950 INFO mapred.MapTask (MapTask.java:createSortingCollector(402)) - Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer
2015-02-12 09:42:33,950 INFO hadoop.FlowMapper (FlowMapper.java:configure(71)) - cascading version: 2.6.3
2015-02-12 09:42:33,950 INFO hadoop.FlowMapper (FlowMapper.java:configure(72)) - child jvm opts: -Xmx200m
2015-02-12 09:42:33,955 INFO hadoop.FlowMapper (FlowMapper.java:configure(87)) - sourcing from: Hfs["TextLine[[0:1]->[ALL]]"]["/Users/soberoi/src/compatibility/cascading.compatibility/horton/build/classes/test/data/ips.20.txt"]
2015-02-12 09:42:33,956 INFO hadoop.FlowMapper (FlowMapper.java:configure(90)) - sinking to: GroupBy(count)[by:[{1}:1]]
2015-02-12 09:42:33,959 INFO mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) -
2015-02-12 09:42:33,959 INFO mapred.MapTask (MapTask.java:flush(1457)) - Starting flush of map output
2015-02-12 09:42:33,959 INFO mapred.MapTask (MapTask.java:flush(1475)) - Spilling map output
2015-02-12 09:42:33,959 INFO mapred.MapTask (MapTask.java:flush(1476)) - bufstart = 0; bufend = 466; bufvoid = 104857600
2015-02-12 09:42:33,959 INFO mapred.MapTask (MapTask.java:flush(1478)) - kvstart = 26214396(104857584); kvend = 26214320(104857280); length = 77/6553600
2015-02-12 09:42:33,963 INFO mapred.MapTask (MapTask.java:sortAndSpill(1660)) - Finished spill 0
2015-02-12 09:42:33,965 INFO mapred.Task (Task.java:done(1001)) - Task:attempt_local1796527190_0011_m_000000_0 is done. And is in the process of committing
2015-02-12 09:42:33,966 INFO mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) - file:/Users/soberoi/src/compatibility/cascading.compatibility/horton/build/classes/test/data/ips.20.txt:0+275
2015-02-12 09:42:33,967 INFO mapred.Task (Task.java:sendDone(1121)) - Task 'attempt_local1796527190_0011_m_000000_0' done.
2015-02-12 09:42:33,967 INFO mapred.LocalJobRunner (LocalJobRunner.java:run(249)) - Finishing task: attempt_local1796527190_0011_m_000000_0
2015-02-12 09:42:33,967 INFO mapred.LocalJobRunner (LocalJobRunner.java:runTasks(456)) - map task executor complete.
2015-02-12 09:42:33,967 INFO mapred.LocalJobRunner (LocalJobRunner.java:runTasks(448)) - Waiting for reduce tasks
2015-02-12 09:42:33,968 INFO mapred.LocalJobRunner (LocalJobRunner.java:run(302)) - Starting task: attempt_local1796527190_0011_r_000000_0
2015-02-12 09:42:33,969 INFO util.ProcfsBasedProcessTree (ProcfsBasedProcessTree.java:isAvailable(181)) - ProcfsBasedProcessTree currently is supported only on Linux.
2015-02-12 09:42:33,969 INFO mapred.Task (Task.java:initialize(587)) - Using ResourceCalculatorProcessTree : null
2015-02-12 09:42:33,970 INFO mapred.ReduceTask (ReduceTask.java:run(362)) - Using ShuffleConsumerPlugin: org.apache.hadoop.mapreduce.task.reduce.Shuffle@5d9c26c8
2015-02-12 09:42:33,970 INFO reduce.MergeManagerImpl (MergeManagerImpl.java:<init>(196)) - MergerManager: memoryLimit=493250144, maxSingleShuffleLimit=123312536, mergeThreshold=325545120, ioSortFactor=10, memToMemMergeOutputsThreshold=10
2015-02-12 09:42:33,971 INFO reduce.EventFetcher (EventFetcher.java:run(61)) - attempt_local1796527190_0011_r_000000_0 Thread started: EventFetcher for fetching Map Completion Events
2015-02-12 09:42:33,972 INFO reduce.LocalFetcher (LocalFetcher.java:copyMapOutput(141)) - localfetcher#7 about to shuffle output of map attempt_local1796527190_0011_m_000000_0 decomp: 508 len: 512 to MEMORY
2015-02-12 09:42:33,973 INFO reduce.InMemoryMapOutput (InMemoryMapOutput.java:shuffle(100)) - Read 508 bytes from map-output for attempt_local1796527190_0011_m_000000_0
2015-02-12 09:42:33,973 INFO reduce.MergeManagerImpl (MergeManagerImpl.java:closeInMemoryFile(314)) - closeInMemoryFile -> map-output of size: 508, inMemoryMapOutputs.size() -> 1, commitMemory -> 0, usedMemory ->508
2015-02-12 09:42:33,973 INFO reduce.EventFetcher (EventFetcher.java:run(76)) - EventFetcher is interrupted.. Returning
2015-02-12 09:42:33,974 INFO mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) - 1 / 1 copied.
2015-02-12 09:42:33,974 INFO reduce.MergeManagerImpl (MergeManagerImpl.java:finalMerge(674)) - finalMerge called with 1 in-memory map-outputs and 0 on-disk map-outputs
2015-02-12 09:42:33,976 INFO mapred.Merger (Merger.java:merge(597)) - Merging 1 sorted segments
2015-02-12 09:42:33,976 INFO mapred.Merger (Merger.java:merge(696)) - Down to the last merge-pass, with 1 segments left of total size: 487 bytes
2015-02-12 09:42:33,977 INFO reduce.MergeManagerImpl (MergeManagerImpl.java:finalMerge(751)) - Merged 1 segments, 508 bytes to disk to satisfy reduce memory limit
2015-02-12 09:42:33,977 INFO reduce.MergeManagerImpl (MergeManagerImpl.java:finalMerge(781)) - Merging 1 files, 512 bytes from disk
2015-02-12 09:42:33,978 INFO reduce.MergeManagerImpl (MergeManagerImpl.java:finalMerge(796)) - Merging 0 segments, 0 bytes from memory into reduce
2015-02-12 09:42:33,978 INFO mapred.Merger (Merger.java:merge(597)) - Merging 1 sorted segments
2015-02-12 09:42:33,978 INFO mapred.Merger (Merger.java:merge(696)) - Down to the last merge-pass, with 1 segments left of total size: 487 bytes
2015-02-12 09:42:33,979 INFO mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) - 1 / 1 copied.
2015-02-12 09:42:33,979 INFO hadoop.FlowReducer (FlowReducer.java:configure(78)) - cascading version: 2.6.3
2015-02-12 09:42:33,979 INFO hadoop.FlowReducer (FlowReducer.java:configure(79)) - child jvm opts: -Xmx200m
2015-02-12 09:42:33,984 INFO hadoop.FlowReducer (FlowReducer.java:configure(97)) - sourcing from: GroupBy(count)[by:[{1}:1]]
2015-02-12 09:42:33,984 INFO hadoop.FlowReducer (FlowReducer.java:configure(100)) - sinking to: Hfs["TextLine[[0]->[ALL]]"]["/Users/soberoi/src/compatibility/cascading.compatibility/horton/build/test/output/hadoop2-mr1/basicpipesplatform/count2"]
2015-02-12 09:42:33,996 INFO mapred.Task (Task.java:done(1001)) - Task:attempt_local1796527190_0011_r_000000_0 is done. And is in the process of committing
2015-02-12 09:42:33,997 INFO mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) - 1 / 1 copied.
2015-02-12 09:42:33,997 INFO mapred.Task (Task.java:commit(1162)) - Task attempt_local1796527190_0011_r_000000_0 is allowed to commit now
2015-02-12 09:42:33,998 INFO output.FileOutputCommitter (FileOutputCommitter.java:commitTask(439)) - Saved output of task 'attempt_local1796527190_0011_r_000000_0' to file:/Users/soberoi/src/compatibility/cascading.compatibility/horton/build/test/output/hadoop2-mr1/basicpipesplatform/count2/_temporary/0/task_local1796527190_0011_r_000000
2015-02-12 09:42:33,999 INFO mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) - reduce > reduce
2015-02-12 09:42:33,999 INFO mapred.Task (Task.java:sendDone(1121)) - Task 'attempt_local1796527190_0011_r_000000_0' done.
2015-02-12 09:42:33,999 INFO mapred.LocalJobRunner (LocalJobRunner.java:run(325)) - Finishing task: attempt_local1796527190_0011_r_000000_0
2015-02-12 09:42:34,000 INFO mapred.LocalJobRunner (LocalJobRunner.java:runTasks(456)) - reduce task executor complete.
2015-02-12 09:42:34,022 INFO util.Hadoop18TapUtil (Hadoop18TapUtil.java:cleanTempPath(219)) - deleting temp path /Users/soberoi/src/compatibility/cascading.compatibility/horton/build/test/output/hadoop2-mr1/basicpipesplatform/count2/_temporary
2015-02-12 09:42:34,056 INFO mapred.FileInputFormat (FileInputFormat.java:listStatus(247)) - Total input paths to process : 1
2015-02-12 09:42:34,090 INFO mapred.FileInputFormat (FileInputFormat.java:listStatus(247)) - Total input paths to process : 1
2015-02-12 09:42:34,092 INFO cascading.PlatformTestCase (PlatformTestCase.java:tearDown(209)) - copying to local /Users/soberoi/src/compatibility/cascading.compatibility/horton/build/test/output//hadoop2-mr1/basicpipesplatform/count2
2015-02-12 09:42:34,113 INFO util.HadoopUtil (HadoopUtil.java:findMainClass(336)) - using default application jar, may cause class not found exceptions on the cluster
2015-02-12 09:42:34,114 INFO planner.HadoopPlanner (HadoopPlanner.java:initialize(225)) - using application jar: /Users/soberoi/.gradle/caches/modules-2/files-2.1/cascading/cascading-hadoop2-mr1/2.6.3/9ae016e2b8d246b9c1de9168e715d9f1b971aa4a/cascading-hadoop2-mr1-2.6.3.jar
2015-02-12 09:42:34,120 INFO Configuration.deprecation (Configuration.java:warnOnceIfDeprecated(1051)) - mapred.used.genericoptionsparser is deprecated. Instead, use mapreduce.client.genericoptionsparser.used
2015-02-12 09:42:34,122 INFO Configuration.deprecation (Configuration.java:warnOnceIfDeprecated(1051)) - mapred.job.tracker is deprecated. Instead, use mapreduce.jobtracker.address
2015-02-12 09:42:34,122 INFO Configuration.deprecation (Configuration.java:warnOnceIfDeprecated(1051)) - mapred.output.compress is deprecated. Instead, use mapreduce.output.fileoutputformat.compress
2015-02-12 09:42:34,129 INFO flow.Flow (BaseFlow.java:logInfo(1378)) - [count] starting
2015-02-12 09:42:34,129 INFO flow.Flow (BaseFlow.java:logInfo(1378)) - [count] source: Hfs["TextLine[[0:1]->[ALL]]"]["/Users/soberoi/src/compatibility/cascading.compatibility/horton/build/classes/test/data/ips.20.txt"]
2015-02-12 09:42:34,130 INFO flow.Flow (BaseFlow.java:logInfo(1378)) - [count] sink: Hfs["TextLine[[0]->[ALL]]"]["/Users/soberoi/src/compatibility/cascading.compatibility/horton/build/test/output/hadoop2-mr1/basicpipesplatform/count3"]
2015-02-12 09:42:34,130 INFO flow.Flow (BaseFlow.java:logInfo(1378)) - [count] parallel execution is enabled: false
2015-02-12 09:42:34,130 INFO flow.Flow (BaseFlow.java:logInfo(1378)) - [count] starting jobs: 1
2015-02-12 09:42:34,130 INFO flow.Flow (BaseFlow.java:logInfo(1378)) - [count] allocating threads: 1
2015-02-12 09:42:34,130 INFO flow.FlowStep (BaseFlowStep.java:logInfo(834)) - [count] starting step: (1/1) ...basicpipesplatform/count3
2015-02-12 09:42:34,131 INFO jvm.JvmMetrics (JvmMetrics.java:init(71)) - Cannot initialize JVM Metrics with processName=JobTracker, sessionId= - already initialized
2015-02-12 09:42:34,133 INFO jvm.JvmMetrics (JvmMetrics.java:init(71)) - Cannot initialize JVM Metrics with processName=JobTracker, sessionId= - already initialized
2015-02-12 09:42:34,159 INFO mapred.FileInputFormat (FileInputFormat.java:listStatus(247)) - Total input paths to process : 1
2015-02-12 09:42:34,179 INFO mapreduce.JobSubmitter (JobSubmitter.java:submitJobInternal(494)) - number of splits:1
2015-02-12 09:42:34,192 INFO mapreduce.JobSubmitter (JobSubmitter.java:printTokens(583)) - Submitting tokens for job: job_local2030413932_0012
2015-02-12 09:42:34,245 INFO mapreduce.Job (Job.java:submit(1300)) - The url to track the job: http://localhost:8080/
2015-02-12 09:42:34,245 INFO mapred.LocalJobRunner (LocalJobRunner.java:createOutputCommitter(471)) - OutputCommitter set in config null
2015-02-12 09:42:34,245 INFO flow.FlowStep (BaseFlowStep.java:logInfo(834)) - [count] submitted hadoop job: job_local2030413932_0012
2015-02-12 09:42:34,245 INFO flow.FlowStep (BaseFlowStep.java:logInfo(834)) - [count] tracking url: http://localhost:8080/
2015-02-12 09:42:34,245 INFO mapred.LocalJobRunner (LocalJobRunner.java:createOutputCommitter(489)) - OutputCommitter is org.apache.hadoop.mapred.FileOutputCommitter
2015-02-12 09:42:34,248 INFO mapred.LocalJobRunner (LocalJobRunner.java:runTasks(448)) - Waiting for map tasks
2015-02-12 09:42:34,248 INFO mapred.LocalJobRunner (LocalJobRunner.java:run(224)) - Starting task: attempt_local2030413932_0012_m_000000_0
2015-02-12 09:42:34,249 INFO util.ProcfsBasedProcessTree (ProcfsBasedProcessTree.java:isAvailable(181)) - ProcfsBasedProcessTree currently is supported only on Linux.
2015-02-12 09:42:34,249 INFO mapred.Task (Task.java:initialize(587)) - Using ResourceCalculatorProcessTree : null
2015-02-12 09:42:34,250 INFO io.MultiInputSplit (MultiInputSplit.java:readFields(161)) - current split input path: file:/Users/soberoi/src/compatibility/cascading.compatibility/horton/build/classes/test/data/ips.20.txt
2015-02-12 09:42:34,250 INFO mapred.MapTask (MapTask.java:updateJobWithSplit(482)) - Processing split: cascading.tap.hadoop.io.MultiInputSplit@633a6ad1
2015-02-12 09:42:34,251 INFO mapred.MapTask (MapTask.java:runOldMapper(436)) - numReduceTasks: 1
2015-02-12 09:42:34,271 INFO mapred.MapTask (MapTask.java:setEquator(1202)) - (EQUATOR) 0 kvi 26214396(104857584)
2015-02-12 09:42:34,271 INFO mapred.MapTask (MapTask.java:init(995)) - mapreduce.task.io.sort.mb: 100
2015-02-12 09:42:34,271 INFO mapred.MapTask (MapTask.java:init(996)) - soft limit at 83886080
2015-02-12 09:42:34,271 INFO mapred.MapTask (MapTask.java:init(997)) - bufstart = 0; bufvoid = 104857600
2015-02-12 09:42:34,271 INFO mapred.MapTask (MapTask.java:init(998)) - kvstart = 26214396; length = 6553600
2015-02-12 09:42:34,272 INFO mapred.MapTask (MapTask.java:createSortingCollector(402)) - Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer
2015-02-12 09:42:34,272 INFO hadoop.FlowMapper (FlowMapper.java:configure(71)) - cascading version: 2.6.3
2015-02-12 09:42:34,272 INFO hadoop.FlowMapper (FlowMapper.java:configure(72)) - child jvm opts: -Xmx200m
2015-02-12 09:42:34,276 INFO hadoop.FlowMapper (FlowMapper.java:configure(87)) - sourcing from: Hfs["TextLine[[0:1]->[ALL]]"]["/Users/soberoi/src/compatibility/cascading.compatibility/horton/build/classes/test/data/ips.20.txt"]
2015-02-12 09:42:34,276 INFO hadoop.FlowMapper (FlowMapper.java:configure(90)) - sinking to: GroupBy(count)[by:[{1}:1]]
2015-02-12 09:42:34,279 INFO mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) -
2015-02-12 09:42:34,279 INFO mapred.MapTask (MapTask.java:flush(1457)) - Starting flush of map output
2015-02-12 09:42:34,279 INFO mapred.MapTask (MapTask.java:flush(1475)) - Spilling map output
2015-02-12 09:42:34,279 INFO mapred.MapTask (MapTask.java:flush(1476)) - bufstart = 0; bufend = 466; bufvoid = 104857600
2015-02-12 09:42:34,280 INFO mapred.MapTask (MapTask.java:flush(1478)) - kvstart = 26214396(104857584); kvend = 26214320(104857280); length = 77/6553600
2015-02-12 09:42:34,282 INFO mapred.MapTask (MapTask.java:sortAndSpill(1660)) - Finished spill 0
2015-02-12 09:42:34,283 INFO mapred.Task (Task.java:done(1001)) - Task:attempt_local2030413932_0012_m_000000_0 is done. And is in the process of committing
2015-02-12 09:42:34,284 INFO mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) - file:/Users/soberoi/src/compatibility/cascading.compatibility/horton/build/classes/test/data/ips.20.txt:0+275
2015-02-12 09:42:34,284 INFO mapred.Task (Task.java:sendDone(1121)) - Task 'attempt_local2030413932_0012_m_000000_0' done.
2015-02-12 09:42:34,284 INFO mapred.LocalJobRunner (LocalJobRunner.java:run(249)) - Finishing task: attempt_local2030413932_0012_m_000000_0
2015-02-12 09:42:34,285 INFO mapred.LocalJobRunner (LocalJobRunner.java:runTasks(456)) - map task executor complete.
2015-02-12 09:42:34,285 INFO mapred.LocalJobRunner (LocalJobRunner.java:runTasks(448)) - Waiting for reduce tasks
2015-02-12 09:42:34,285 INFO mapred.LocalJobRunner (LocalJobRunner.java:run(302)) - Starting task: attempt_local2030413932_0012_r_000000_0
2015-02-12 09:42:34,286 INFO util.ProcfsBasedProcessTree (ProcfsBasedProcessTree.java:isAvailable(181)) - ProcfsBasedProcessTree currently is supported only on Linux.
2015-02-12 09:42:34,286 INFO mapred.Task (Task.java:initialize(587)) - Using ResourceCalculatorProcessTree : null
2015-02-12 09:42:34,286 INFO mapred.ReduceTask (ReduceTask.java:run(362)) - Using ShuffleConsumerPlugin: org.apache.hadoop.mapreduce.task.reduce.Shuffle@2c8cd466
2015-02-12 09:42:34,287 INFO reduce.MergeManagerImpl (MergeManagerImpl.java:<init>(196)) - MergerManager: memoryLimit=493250144, maxSingleShuffleLimit=123312536, mergeThreshold=325545120, ioSortFactor=10, memToMemMergeOutputsThreshold=10
2015-02-12 09:42:34,287 INFO reduce.EventFetcher (EventFetcher.java:run(61)) - attempt_local2030413932_0012_r_000000_0 Thread started: EventFetcher for fetching Map Completion Events
2015-02-12 09:42:34,289 INFO reduce.LocalFetcher (LocalFetcher.java:copyMapOutput(141)) - localfetcher#8 about to shuffle output of map attempt_local2030413932_0012_m_000000_0 decomp: 508 len: 512 to MEMORY
2015-02-12 09:42:34,289 INFO reduce.InMemoryMapOutput (InMemoryMapOutput.java:shuffle(100)) - Read 508 bytes from map-output for attempt_local2030413932_0012_m_000000_0
2015-02-12 09:42:34,289 INFO reduce.MergeManagerImpl (MergeManagerImpl.java:closeInMemoryFile(314)) - closeInMemoryFile -> map-output of size: 508, inMemoryMapOutputs.size() -> 1, commitMemory -> 0, usedMemory ->508
2015-02-12 09:42:34,289 INFO reduce.EventFetcher (EventFetcher.java:run(76)) - EventFetcher is interrupted.. Returning
2015-02-12 09:42:34,290 INFO mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) - 1 / 1 copied.
2015-02-12 09:42:34,290 INFO reduce.MergeManagerImpl (MergeManagerImpl.java:finalMerge(674)) - finalMerge called with 1 in-memory map-outputs and 0 on-disk map-outputs
2015-02-12 09:42:34,292 INFO mapred.Merger (Merger.java:merge(597)) - Merging 1 sorted segments
2015-02-12 09:42:34,292 INFO mapred.Merger (Merger.java:merge(696)) - Down to the last merge-pass, with 1 segments left of total size: 487 bytes
2015-02-12 09:42:34,293 INFO reduce.MergeManagerImpl (MergeManagerImpl.java:finalMerge(751)) - Merged 1 segments, 508 bytes to disk to satisfy reduce memory limit
2015-02-12 09:42:34,293 INFO reduce.MergeManagerImpl (MergeManagerImpl.java:finalMerge(781)) - Merging 1 files, 512 bytes from disk
2015-02-12 09:42:34,293 INFO reduce.MergeManagerImpl (MergeManagerImpl.java:finalMerge(796)) - Merging 0 segments, 0 bytes from memory into reduce
2015-02-12 09:42:34,294 INFO mapred.Merger (Merger.java:merge(597)) - Merging 1 sorted segments
2015-02-12 09:42:34,294 INFO mapred.Merger (Merger.java:merge(696)) - Down to the last merge-pass, with 1 segments left of total size: 487 bytes
2015-02-12 09:42:34,295 INFO mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) - 1 / 1 copied.
2015-02-12 09:42:34,295 INFO hadoop.FlowReducer (FlowReducer.java:configure(78)) - cascading version: 2.6.3
2015-02-12 09:42:34,295 INFO hadoop.FlowReducer (FlowReducer.java:configure(79)) - child jvm opts: -Xmx200m
2015-02-12 09:42:34,299 INFO hadoop.FlowReducer (FlowReducer.java:configure(97)) - sourcing from: GroupBy(count)[by:[{1}:1]]
2015-02-12 09:42:34,299 INFO hadoop.FlowReducer (FlowReducer.java:configure(100)) - sinking to: Hfs["TextLine[[0]->[ALL]]"]["/Users/soberoi/src/compatibility/cascading.compatibility/horton/build/test/output/hadoop2-mr1/basicpipesplatform/count3"]
2015-02-12 09:42:34,308 INFO mapred.Task (Task.java:done(1001)) - Task:attempt_local2030413932_0012_r_000000_0 is done. And is in the process of committing
2015-02-12 09:42:34,309 INFO mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) - 1 / 1 copied.
2015-02-12 09:42:34,309 INFO mapred.Task (Task.java:commit(1162)) - Task attempt_local2030413932_0012_r_000000_0 is allowed to commit now
2015-02-12 09:42:34,310 INFO output.FileOutputCommitter (FileOutputCommitter.java:commitTask(439)) - Saved output of task 'attempt_local2030413932_0012_r_000000_0' to file:/Users/soberoi/src/compatibility/cascading.compatibility/horton/build/test/output/hadoop2-mr1/basicpipesplatform/count3/_temporary/0/task_local2030413932_0012_r_000000
2015-02-12 09:42:34,311 INFO mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) - reduce > reduce
2015-02-12 09:42:34,311 INFO mapred.Task (Task.java:sendDone(1121)) - Task 'attempt_local2030413932_0012_r_000000_0' done.
2015-02-12 09:42:34,311 INFO mapred.LocalJobRunner (LocalJobRunner.java:run(325)) - Finishing task: attempt_local2030413932_0012_r_000000_0
2015-02-12 09:42:34,311 INFO mapred.LocalJobRunner (LocalJobRunner.java:runTasks(456)) - reduce task executor complete.
2015-02-12 09:42:34,330 INFO util.Hadoop18TapUtil (Hadoop18TapUtil.java:cleanTempPath(219)) - deleting temp path /Users/soberoi/src/compatibility/cascading.compatibility/horton/build/test/output/hadoop2-mr1/basicpipesplatform/count3/_temporary
2015-02-12 09:42:34,352 INFO mapred.FileInputFormat (FileInputFormat.java:listStatus(247)) - Total input paths to process : 1
2015-02-12 09:42:34,378 INFO mapred.FileInputFormat (FileInputFormat.java:listStatus(247)) - Total input paths to process : 1
2015-02-12 09:42:34,380 INFO cascading.PlatformTestCase (PlatformTestCase.java:tearDown(209)) - copying to local /Users/soberoi/src/compatibility/cascading.compatibility/horton/build/test/output//hadoop2-mr1/basicpipesplatform/count3
2015-02-12 09:42:34,394 INFO util.HadoopUtil (HadoopUtil.java:findMainClass(336)) - using default application jar, may cause class not found exceptions on the cluster
2015-02-12 09:42:34,394 INFO planner.HadoopPlanner (HadoopPlanner.java:initialize(225)) - using application jar: /Users/soberoi/.gradle/caches/modules-2/files-2.1/cascading/cascading-hadoop2-mr1/2.6.3/9ae016e2b8d246b9c1de9168e715d9f1b971aa4a/cascading-hadoop2-mr1-2.6.3.jar
2015-02-12 09:42:34,399 INFO Configuration.deprecation (Configuration.java:warnOnceIfDeprecated(1051)) - mapred.used.genericoptionsparser is deprecated. Instead, use mapreduce.client.genericoptionsparser.used
2015-02-12 09:42:34,400 INFO Configuration.deprecation (Configuration.java:warnOnceIfDeprecated(1051)) - mapred.job.tracker is deprecated. Instead, use mapreduce.jobtracker.address
2015-02-12 09:42:34,401 INFO Configuration.deprecation (Configuration.java:warnOnceIfDeprecated(1051)) - mapred.output.compress is deprecated. Instead, use mapreduce.output.fileoutputformat.compress
2015-02-12 09:42:34,405 INFO flow.Flow (BaseFlow.java:logInfo(1378)) - [count] starting
2015-02-12 09:42:34,406 INFO flow.Flow (BaseFlow.java:logInfo(1378)) - [count] source: Hfs["TextLine[[0:1]->[ALL]]"]["/Users/soberoi/src/compatibility/cascading.compatibility/horton/build/classes/test/data/ips.20.txt"]
2015-02-12 09:42:34,406 INFO flow.Flow (BaseFlow.java:logInfo(1378)) - [count] sink: Hfs["TextLine[[0]->[ALL]]"]["/Users/soberoi/src/compatibility/cascading.compatibility/horton/build/test/output/hadoop2-mr1/basicpipesplatform/count4"]
2015-02-12 09:42:34,406 INFO flow.Flow (BaseFlow.java:logInfo(1378)) - [count] parallel execution is enabled: false
2015-02-12 09:42:34,406 INFO flow.Flow (BaseFlow.java:logInfo(1378)) - [count] starting jobs: 1
2015-02-12 09:42:34,406 INFO flow.Flow (BaseFlow.java:logInfo(1378)) - [count] allocating threads: 1
2015-02-12 09:42:34,407 INFO flow.FlowStep (BaseFlowStep.java:logInfo(834)) - [count] starting step: (1/1) ...basicpipesplatform/count4
2015-02-12 09:42:34,407 INFO jvm.JvmMetrics (JvmMetrics.java:init(71)) - Cannot initialize JVM Metrics with processName=JobTracker, sessionId= - already initialized
2015-02-12 09:42:34,409 INFO jvm.JvmMetrics (JvmMetrics.java:init(71)) - Cannot initialize JVM Metrics with processName=JobTracker, sessionId= - already initialized
2015-02-12 09:42:34,428 INFO mapred.FileInputFormat (FileInputFormat.java:listStatus(247)) - Total input paths to process : 1
2015-02-12 09:42:34,445 INFO mapreduce.JobSubmitter (JobSubmitter.java:submitJobInternal(494)) - number of splits:1
2015-02-12 09:42:34,457 INFO mapreduce.JobSubmitter (JobSubmitter.java:printTokens(583)) - Submitting tokens for job: job_local1591697442_0013
2015-02-12 09:42:34,504 INFO mapreduce.Job (Job.java:submit(1300)) - The url to track the job: http://localhost:8080/
2015-02-12 09:42:34,505 INFO mapred.LocalJobRunner (LocalJobRunner.java:createOutputCommitter(471)) - OutputCommitter set in config null
2015-02-12 09:42:34,505 INFO flow.FlowStep (BaseFlowStep.java:logInfo(834)) - [count] submitted hadoop job: job_local1591697442_0013
2015-02-12 09:42:34,505 INFO mapred.LocalJobRunner (LocalJobRunner.java:createOutputCommitter(489)) - OutputCommitter is org.apache.hadoop.mapred.FileOutputCommitter
2015-02-12 09:42:34,505 INFO flow.FlowStep (BaseFlowStep.java:logInfo(834)) - [count] tracking url: http://localhost:8080/
2015-02-12 09:42:34,507 INFO mapred.LocalJobRunner (LocalJobRunner.java:runTasks(448)) - Waiting for map tasks
2015-02-12 09:42:34,507 INFO mapred.LocalJobRunner (LocalJobRunner.java:run(224)) - Starting task: attempt_local1591697442_0013_m_000000_0
2015-02-12 09:42:34,508 INFO util.ProcfsBasedProcessTree (ProcfsBasedProcessTree.java:isAvailable(181)) - ProcfsBasedProcessTree currently is supported only on Linux.
2015-02-12 09:42:34,508 INFO mapred.Task (Task.java:initialize(587)) - Using ResourceCalculatorProcessTree : null
2015-02-12 09:42:34,509 INFO io.MultiInputSplit (MultiInputSplit.java:readFields(161)) - current split input path: file:/Users/soberoi/src/compatibility/cascading.compatibility/horton/build/classes/test/data/ips.20.txt
2015-02-12 09:42:34,509 INFO mapred.MapTask (MapTask.java:updateJobWithSplit(482)) - Processing split: cascading.tap.hadoop.io.MultiInputSplit@264348eb
2015-02-12 09:42:34,510 INFO mapred.MapTask (MapTask.java:runOldMapper(436)) - numReduceTasks: 1
2015-02-12 09:42:34,530 INFO mapred.MapTask (MapTask.java:setEquator(1202)) - (EQUATOR) 0 kvi 26214396(104857584)
2015-02-12 09:42:34,530 INFO mapred.MapTask (MapTask.java:init(995)) - mapreduce.task.io.sort.mb: 100
2015-02-12 09:42:34,530 INFO mapred.MapTask (MapTask.java:init(996)) - soft limit at 83886080
2015-02-12 09:42:34,530 INFO mapred.MapTask (MapTask.java:init(997)) - bufstart = 0; bufvoid = 104857600
2015-02-12 09:42:34,530 INFO mapred.MapTask (MapTask.java:init(998)) - kvstart = 26214396; length = 6553600
2015-02-12 09:42:34,531 INFO mapred.MapTask (MapTask.java:createSortingCollector(402)) - Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer
2015-02-12 09:42:34,531 INFO hadoop.FlowMapper (FlowMapper.java:configure(71)) - cascading version: 2.6.3
2015-02-12 09:42:34,532 INFO hadoop.FlowMapper (FlowMapper.java:configure(72)) - child jvm opts: -Xmx200m
2015-02-12 09:42:34,535 INFO hadoop.FlowMapper (FlowMapper.java:configure(87)) - sourcing from: Hfs["TextLine[[0:1]->[ALL]]"]["/Users/soberoi/src/compatibility/cascading.compatibility/horton/build/classes/test/data/ips.20.txt"]
2015-02-12 09:42:34,535 INFO hadoop.FlowMapper (FlowMapper.java:configure(90)) - sinking to: GroupBy(count)[by:[{1}:1]]
2015-02-12 09:42:34,537 INFO mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) -
2015-02-12 09:42:34,537 INFO mapred.MapTask (MapTask.java:flush(1457)) - Starting flush of map output
2015-02-12 09:42:34,537 INFO mapred.MapTask (MapTask.java:flush(1475)) - Spilling map output
2015-02-12 09:42:34,537 INFO mapred.MapTask (MapTask.java:flush(1476)) - bufstart = 0; bufend = 466; bufvoid = 104857600
2015-02-12 09:42:34,538 INFO mapred.MapTask (MapTask.java:flush(1478)) - kvstart = 26214396(104857584); kvend = 26214320(104857280); length = 77/6553600
2015-02-12 09:42:34,540 INFO mapred.MapTask (MapTask.java:sortAndSpill(1660)) - Finished spill 0
2015-02-12 09:42:34,541 INFO mapred.Task (Task.java:done(1001)) - Task:attempt_local1591697442_0013_m_000000_0 is done. And is in the process of committing
2015-02-12 09:42:34,543 INFO mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) - file:/Users/soberoi/src/compatibility/cascading.compatibility/horton/build/classes/test/data/ips.20.txt:0+275
2015-02-12 09:42:34,543 INFO mapred.Task (Task.java:sendDone(1121)) - Task 'attempt_local1591697442_0013_m_000000_0' done.
2015-02-12 09:42:34,543 INFO mapred.LocalJobRunner (LocalJobRunner.java:run(249)) - Finishing task: attempt_local1591697442_0013_m_000000_0
2015-02-12 09:42:34,543 INFO mapred.LocalJobRunner (LocalJobRunner.java:runTasks(456)) - map task executor complete.
2015-02-12 09:42:34,543 INFO mapred.LocalJobRunner (LocalJobRunner.java:runTasks(448)) - Waiting for reduce tasks
2015-02-12 09:42:34,544 INFO mapred.LocalJobRunner (LocalJobRunner.java:run(302)) - Starting task: attempt_local1591697442_0013_r_000000_0
2015-02-12 09:42:34,545 INFO util.ProcfsBasedProcessTree (ProcfsBasedProcessTree.java:isAvailable(181)) - ProcfsBasedProcessTree currently is supported only on Linux.
2015-02-12 09:42:34,545 INFO mapred.Task (Task.java:initialize(587)) - Using ResourceCalculatorProcessTree : null
2015-02-12 09:42:34,545 INFO mapred.ReduceTask (ReduceTask.java:run(362)) - Using ShuffleConsumerPlugin: org.apache.hadoop.mapreduce.task.reduce.Shuffle@7b41d5c5
2015-02-12 09:42:34,546 INFO reduce.MergeManagerImpl (MergeManagerImpl.java:<init>(196)) - MergerManager: memoryLimit=493250144, maxSingleShuffleLimit=123312536, mergeThreshold=325545120, ioSortFactor=10, memToMemMergeOutputsThreshold=10
2015-02-12 09:42:34,546 INFO reduce.EventFetcher (EventFetcher.java:run(61)) - attempt_local1591697442_0013_r_000000_0 Thread started: EventFetcher for fetching Map Completion Events
2015-02-12 09:42:34,547 INFO reduce.LocalFetcher (LocalFetcher.java:copyMapOutput(141)) - localfetcher#9 about to shuffle output of map attempt_local1591697442_0013_m_000000_0 decomp: 508 len: 512 to MEMORY
2015-02-12 09:42:34,548 INFO reduce.InMemoryMapOutput (InMemoryMapOutput.java:shuffle(100)) - Read 508 bytes from map-output for attempt_local1591697442_0013_m_000000_0
2015-02-12 09:42:34,548 INFO reduce.MergeManagerImpl (MergeManagerImpl.java:closeInMemoryFile(314)) - closeInMemoryFile -> map-output of size: 508, inMemoryMapOutputs.size() -> 1, commitMemory -> 0, usedMemory ->508
2015-02-12 09:42:34,548 INFO reduce.EventFetcher (EventFetcher.java:run(76)) - EventFetcher is interrupted.. Returning
2015-02-12 09:42:34,549 INFO mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) - 1 / 1 copied.
2015-02-12 09:42:34,549 INFO reduce.MergeManagerImpl (MergeManagerImpl.java:finalMerge(674)) - finalMerge called with 1 in-memory map-outputs and 0 on-disk map-outputs
2015-02-12 09:42:34,551 INFO mapred.Merger (Merger.java:merge(597)) - Merging 1 sorted segments
2015-02-12 09:42:34,551 INFO mapred.Merger (Merger.java:merge(696)) - Down to the last merge-pass, with 1 segments left of total size: 487 bytes
2015-02-12 09:42:34,552 INFO reduce.MergeManagerImpl (MergeManagerImpl.java:finalMerge(751)) - Merged 1 segments, 508 bytes to disk to satisfy reduce memory limit
2015-02-12 09:42:34,552 INFO reduce.MergeManagerImpl (MergeManagerImpl.java:finalMerge(781)) - Merging 1 files, 512 bytes from disk
2015-02-12 09:42:34,552 INFO reduce.MergeManagerImpl (MergeManagerImpl.java:finalMerge(796)) - Merging 0 segments, 0 bytes from memory into reduce
2015-02-12 09:42:34,552 INFO mapred.Merger (Merger.java:merge(597)) - Merging 1 sorted segments
2015-02-12 09:42:34,553 INFO mapred.Merger (Merger.java:merge(696)) - Down to the last merge-pass, with 1 segments left of total size: 487 bytes
2015-02-12 09:42:34,553 INFO mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) - 1 / 1 copied.
2015-02-12 09:42:34,554 INFO hadoop.FlowReducer (FlowReducer.java:configure(78)) - cascading version: 2.6.3
2015-02-12 09:42:34,554 INFO hadoop.FlowReducer (FlowReducer.java:configure(79)) - child jvm opts: -Xmx200m
2015-02-12 09:42:34,559 INFO hadoop.FlowReducer (FlowReducer.java:configure(97)) - sourcing from: GroupBy(count)[by:[{1}:1]]
2015-02-12 09:42:34,559 INFO hadoop.FlowReducer (FlowReducer.java:configure(100)) - sinking to: Hfs["TextLine[[0]->[ALL]]"]["/Users/soberoi/src/compatibility/cascading.compatibility/horton/build/test/output/hadoop2-mr1/basicpipesplatform/count4"]
2015-02-12 09:42:34,570 INFO mapred.Task (Task.java:done(1001)) - Task:attempt_local1591697442_0013_r_000000_0 is done. And is in the process of committing
2015-02-12 09:42:34,572 INFO mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) - 1 / 1 copied.
2015-02-12 09:42:34,572 INFO mapred.Task (Task.java:commit(1162)) - Task attempt_local1591697442_0013_r_000000_0 is allowed to commit now
2015-02-12 09:42:34,573 INFO output.FileOutputCommitter (FileOutputCommitter.java:commitTask(439)) - Saved output of task 'attempt_local1591697442_0013_r_000000_0' to file:/Users/soberoi/src/compatibility/cascading.compatibility/horton/build/test/output/hadoop2-mr1/basicpipesplatform/count4/_temporary/0/task_local1591697442_0013_r_000000
2015-02-12 09:42:34,574 INFO mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) - reduce > reduce
2015-02-12 09:42:34,574 INFO mapred.Task (Task.java:sendDone(1121)) - Task 'attempt_local1591697442_0013_r_000000_0' done.
2015-02-12 09:42:34,575 INFO mapred.LocalJobRunner (LocalJobRunner.java:run(325)) - Finishing task: attempt_local1591697442_0013_r_000000_0
2015-02-12 09:42:34,575 INFO mapred.LocalJobRunner (LocalJobRunner.java:runTasks(456)) - reduce task executor complete.
2015-02-12 09:42:34,592 INFO util.Hadoop18TapUtil (Hadoop18TapUtil.java:cleanTempPath(219)) - deleting temp path /Users/soberoi/src/compatibility/cascading.compatibility/horton/build/test/output/hadoop2-mr1/basicpipesplatform/count4/_temporary
2015-02-12 09:42:34,616 INFO mapred.FileInputFormat (FileInputFormat.java:listStatus(247)) - Total input paths to process : 1
2015-02-12 09:42:34,639 INFO mapred.FileInputFormat (FileInputFormat.java:listStatus(247)) - Total input paths to process : 1
2015-02-12 09:42:34,640 INFO cascading.PlatformTestCase (PlatformTestCase.java:tearDown(209)) - copying to local /Users/soberoi/src/compatibility/cascading.compatibility/horton/build/test/output//hadoop2-mr1/basicpipesplatform/count4
2015-02-12 09:42:34,651 INFO util.HadoopUtil (HadoopUtil.java:findMainClass(336)) - using default application jar, may cause class not found exceptions on the cluster
2015-02-12 09:42:34,652 INFO planner.HadoopPlanner (HadoopPlanner.java:initialize(225)) - using application jar: /Users/soberoi/.gradle/caches/modules-2/files-2.1/cascading/cascading-hadoop2-mr1/2.6.3/9ae016e2b8d246b9c1de9168e715d9f1b971aa4a/cascading-hadoop2-mr1-2.6.3.jar
2015-02-12 09:42:34,655 INFO Configuration.deprecation (Configuration.java:warnOnceIfDeprecated(1051)) - mapred.used.genericoptionsparser is deprecated. Instead, use mapreduce.client.genericoptionsparser.used
2015-02-12 09:42:34,657 INFO Configuration.deprecation (Configuration.java:warnOnceIfDeprecated(1051)) - mapred.job.tracker is deprecated. Instead, use mapreduce.jobtracker.address
2015-02-12 09:42:34,657 INFO Configuration.deprecation (Configuration.java:warnOnceIfDeprecated(1051)) - mapred.output.compress is deprecated. Instead, use mapreduce.output.fileoutputformat.compress
2015-02-12 09:42:34,662 INFO flow.Flow (BaseFlow.java:logInfo(1378)) - [test] starting
2015-02-12 09:42:34,662 INFO flow.Flow (BaseFlow.java:logInfo(1378)) - [test] source: Hfs["TextLine[[0:1]->[ALL]]"]["/Users/soberoi/src/compatibility/cascading.compatibility/horton/build/classes/test/data/apache.10.txt"]
2015-02-12 09:42:34,662 INFO flow.Flow (BaseFlow.java:logInfo(1378)) - [test] sink: Hfs["TextLine[['offset', 'line']->[ALL]]"]["/Users/soberoi/src/compatibility/cascading.compatibility/horton/build/test/output/hadoop2-mr1/basicpipesplatform/filter"]
2015-02-12 09:42:34,662 INFO flow.Flow (BaseFlow.java:logInfo(1378)) - [test] parallel execution is enabled: false
2015-02-12 09:42:34,662 INFO flow.Flow (BaseFlow.java:logInfo(1378)) - [test] starting jobs: 1
2015-02-12 09:42:34,662 INFO flow.Flow (BaseFlow.java:logInfo(1378)) - [test] allocating threads: 1
2015-02-12 09:42:34,663 INFO flow.FlowStep (BaseFlowStep.java:logInfo(834)) - [test] starting step: (1/1) ...basicpipesplatform/filter
2015-02-12 09:42:34,663 INFO jvm.JvmMetrics (JvmMetrics.java:init(71)) - Cannot initialize JVM Metrics with processName=JobTracker, sessionId= - already initialized
2015-02-12 09:42:34,664 INFO jvm.JvmMetrics (JvmMetrics.java:init(71)) - Cannot initialize JVM Metrics with processName=JobTracker, sessionId= - already initialized
2015-02-12 09:42:34,685 INFO mapred.FileInputFormat (FileInputFormat.java:listStatus(247)) - Total input paths to process : 1
2015-02-12 09:42:34,700 INFO mapreduce.JobSubmitter (JobSubmitter.java:submitJobInternal(494)) - number of splits:1
2015-02-12 09:42:34,712 INFO mapreduce.JobSubmitter (JobSubmitter.java:printTokens(583)) - Submitting tokens for job: job_local1029960701_0014
2015-02-12 09:42:34,755 INFO mapreduce.Job (Job.java:submit(1300)) - The url to track the job: http://localhost:8080/
2015-02-12 09:42:34,755 INFO mapred.LocalJobRunner (LocalJobRunner.java:createOutputCommitter(471)) - OutputCommitter set in config null
2015-02-12 09:42:34,755 INFO flow.FlowStep (BaseFlowStep.java:logInfo(834)) - [test] submitted hadoop job: job_local1029960701_0014
2015-02-12 09:42:34,755 INFO flow.FlowStep (BaseFlowStep.java:logInfo(834)) - [test] tracking url: http://localhost:8080/
2015-02-12 09:42:34,755 INFO mapred.LocalJobRunner (LocalJobRunner.java:createOutputCommitter(489)) - OutputCommitter is org.apache.hadoop.mapred.FileOutputCommitter
2015-02-12 09:42:34,757 INFO mapred.LocalJobRunner (LocalJobRunner.java:runTasks(448)) - Waiting for map tasks
2015-02-12 09:42:34,757 INFO mapred.LocalJobRunner (LocalJobRunner.java:run(224)) - Starting task: attempt_local1029960701_0014_m_000000_0
2015-02-12 09:42:34,758 INFO util.ProcfsBasedProcessTree (ProcfsBasedProcessTree.java:isAvailable(181)) - ProcfsBasedProcessTree currently is supported only on Linux.
2015-02-12 09:42:34,759 INFO mapred.Task (Task.java:initialize(587)) - Using ResourceCalculatorProcessTree : null
2015-02-12 09:42:34,759 INFO io.MultiInputSplit (MultiInputSplit.java:readFields(161)) - current split input path: file:/Users/soberoi/src/compatibility/cascading.compatibility/horton/build/classes/test/data/apache.10.txt
2015-02-12 09:42:34,760 INFO mapred.MapTask (MapTask.java:updateJobWithSplit(482)) - Processing split: cascading.tap.hadoop.io.MultiInputSplit@7253ca2f
2015-02-12 09:42:34,760 INFO mapred.MapTask (MapTask.java:runOldMapper(436)) - numReduceTasks: 0
2015-02-12 09:42:34,765 INFO hadoop.FlowMapper (FlowMapper.java:configure(71)) - cascading version: 2.6.3
2015-02-12 09:42:34,765 INFO hadoop.FlowMapper (FlowMapper.java:configure(72)) - child jvm opts: -Xmx200m
2015-02-12 09:42:34,769 INFO hadoop.FlowMapper (FlowMapper.java:configure(87)) - sourcing from: Hfs["TextLine[[0:1]->[ALL]]"]["/Users/soberoi/src/compatibility/cascading.compatibility/horton/build/classes/test/data/apache.10.txt"]
2015-02-12 09:42:34,769 INFO hadoop.FlowMapper (FlowMapper.java:configure(90)) - sinking to: Hfs["TextLine[['offset', 'line']->[ALL]]"]["/Users/soberoi/src/compatibility/cascading.compatibility/horton/build/test/output/hadoop2-mr1/basicpipesplatform/filter"]
2015-02-12 09:42:34,771 INFO mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) -
2015-02-12 09:42:34,771 INFO mapred.Task (Task.java:done(1001)) - Task:attempt_local1029960701_0014_m_000000_0 is done. And is in the process of committing
2015-02-12 09:42:34,772 INFO mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) -
2015-02-12 09:42:34,772 INFO mapred.Task (Task.java:commit(1162)) - Task attempt_local1029960701_0014_m_000000_0 is allowed to commit now
2015-02-12 09:42:34,773 INFO output.FileOutputCommitter (FileOutputCommitter.java:commitTask(439)) - Saved output of task 'attempt_local1029960701_0014_m_000000_0' to file:/Users/soberoi/src/compatibility/cascading.compatibility/horton/build/test/output/hadoop2-mr1/basicpipesplatform/filter/_temporary/0/task_local1029960701_0014_m_000000
2015-02-12 09:42:34,774 INFO mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) - file:/Users/soberoi/src/compatibility/cascading.compatibility/horton/build/classes/test/data/apache.10.txt:0+1449
2015-02-12 09:42:34,774 INFO mapred.Task (Task.java:sendDone(1121)) - Task 'attempt_local1029960701_0014_m_000000_0' done.
2015-02-12 09:42:34,774 INFO mapred.LocalJobRunner (LocalJobRunner.java:run(249)) - Finishing task: attempt_local1029960701_0014_m_000000_0
2015-02-12 09:42:34,774 INFO mapred.LocalJobRunner (LocalJobRunner.java:runTasks(456)) - map task executor complete.
2015-02-12 09:42:34,794 INFO util.Hadoop18TapUtil (Hadoop18TapUtil.java:cleanTempPath(219)) - deleting temp path /Users/soberoi/src/compatibility/cascading.compatibility/horton/build/test/output/hadoop2-mr1/basicpipesplatform/filter/_temporary
2015-02-12 09:42:34,831 INFO mapred.FileInputFormat (FileInputFormat.java:listStatus(247)) - Total input paths to process : 1
2015-02-12 09:42:34,832 INFO cascading.PlatformTestCase (PlatformTestCase.java:tearDown(209)) - copying to local /Users/soberoi/src/compatibility/cascading.compatibility/horton/build/test/output//hadoop2-mr1/basicpipesplatform/filter
2015-02-12 09:42:34,847 INFO util.HadoopUtil (HadoopUtil.java:findMainClass(336)) - using default application jar, may cause class not found exceptions on the cluster
2015-02-12 09:42:34,848 INFO planner.HadoopPlanner (HadoopPlanner.java:initialize(225)) - using application jar: /Users/soberoi/.gradle/caches/modules-2/files-2.1/cascading/cascading-hadoop2-mr1/2.6.3/9ae016e2b8d246b9c1de9168e715d9f1b971aa4a/cascading-hadoop2-mr1-2.6.3.jar
2015-02-12 09:42:34,854 INFO Configuration.deprecation (Configuration.java:warnOnceIfDeprecated(1051)) - mapred.used.genericoptionsparser is deprecated. Instead, use mapreduce.client.genericoptionsparser.used
2015-02-12 09:42:34,856 INFO Configuration.deprecation (Configuration.java:warnOnceIfDeprecated(1051)) - mapred.job.tracker is deprecated. Instead, use mapreduce.jobtracker.address
2015-02-12 09:42:34,857 INFO Configuration.deprecation (Configuration.java:warnOnceIfDeprecated(1051)) - mapred.output.compress is deprecated. Instead, use mapreduce.output.fileoutputformat.compress
2015-02-12 09:42:34,863 INFO flow.Flow (BaseFlow.java:logInfo(1378)) - [test] starting
2015-02-12 09:42:34,864 INFO flow.Flow (BaseFlow.java:logInfo(1378)) - [test] source: Hfs["TextLine[[0:1]->[ALL]]"]["/Users/soberoi/src/compatibility/cascading.compatibility/horton/build/classes/test/data/apache.10.txt"]
2015-02-12 09:42:34,864 INFO flow.Flow (BaseFlow.java:logInfo(1378)) - [test] sink: Hfs["TextLine[[0]->[ALL]]"]["/Users/soberoi/src/compatibility/cascading.compatibility/horton/build/test/output/hadoop2-mr1/basicpipesplatform/simple"]
2015-02-12 09:42:34,864 INFO flow.Flow (BaseFlow.java:logInfo(1378)) - [test] parallel execution is enabled: false
2015-02-12 09:42:34,864 INFO flow.Flow (BaseFlow.java:logInfo(1378)) - [test] starting jobs: 1
2015-02-12 09:42:34,864 INFO flow.Flow (BaseFlow.java:logInfo(1378)) - [test] allocating threads: 1
2015-02-12 09:42:34,864 INFO flow.FlowStep (BaseFlowStep.java:logInfo(834)) - [test] starting step: (1/1) ...basicpipesplatform/simple
2015-02-12 09:42:34,865 INFO jvm.JvmMetrics (JvmMetrics.java:init(71)) - Cannot initialize JVM Metrics with processName=JobTracker, sessionId= - already initialized
2015-02-12 09:42:34,866 INFO jvm.JvmMetrics (JvmMetrics.java:init(71)) - Cannot initialize JVM Metrics with processName=JobTracker, sessionId= - already initialized
2015-02-12 09:42:34,888 INFO mapred.FileInputFormat (FileInputFormat.java:listStatus(247)) - Total input paths to process : 1
2015-02-12 09:42:34,905 INFO mapreduce.JobSubmitter (JobSubmitter.java:submitJobInternal(494)) - number of splits:1
2015-02-12 09:42:34,917 INFO mapreduce.JobSubmitter (JobSubmitter.java:printTokens(583)) - Submitting tokens for job: job_local1186336270_0015
2015-02-12 09:42:34,966 INFO mapreduce.Job (Job.java:submit(1300)) - The url to track the job: http://localhost:8080/
2015-02-12 09:42:34,966 INFO mapred.LocalJobRunner (LocalJobRunner.java:createOutputCommitter(471)) - OutputCommitter set in config null
2015-02-12 09:42:34,966 INFO flow.FlowStep (BaseFlowStep.java:logInfo(834)) - [test] submitted hadoop job: job_local1186336270_0015
2015-02-12 09:42:34,966 INFO mapred.LocalJobRunner (LocalJobRunner.java:createOutputCommitter(489)) - OutputCommitter is org.apache.hadoop.mapred.FileOutputCommitter
2015-02-12 09:42:34,966 INFO flow.FlowStep (BaseFlowStep.java:logInfo(834)) - [test] tracking url: http://localhost:8080/
2015-02-12 09:42:34,968 INFO mapred.LocalJobRunner (LocalJobRunner.java:runTasks(448)) - Waiting for map tasks
2015-02-12 09:42:34,968 INFO mapred.LocalJobRunner (LocalJobRunner.java:run(224)) - Starting task: attempt_local1186336270_0015_m_000000_0
2015-02-12 09:42:34,969 INFO util.ProcfsBasedProcessTree (ProcfsBasedProcessTree.java:isAvailable(181)) - ProcfsBasedProcessTree currently is supported only on Linux.
2015-02-12 09:42:34,969 INFO mapred.Task (Task.java:initialize(587)) - Using ResourceCalculatorProcessTree : null
2015-02-12 09:42:34,971 INFO io.MultiInputSplit (MultiInputSplit.java:readFields(161)) - current split input path: file:/Users/soberoi/src/compatibility/cascading.compatibility/horton/build/classes/test/data/apache.10.txt
2015-02-12 09:42:34,971 INFO mapred.MapTask (MapTask.java:updateJobWithSplit(482)) - Processing split: cascading.tap.hadoop.io.MultiInputSplit@34580d5b
2015-02-12 09:42:34,971 INFO mapred.MapTask (MapTask.java:runOldMapper(436)) - numReduceTasks: 1
2015-02-12 09:42:34,991 INFO mapred.MapTask (MapTask.java:setEquator(1202)) - (EQUATOR) 0 kvi 26214396(104857584)
2015-02-12 09:42:34,991 INFO mapred.MapTask (MapTask.java:init(995)) - mapreduce.task.io.sort.mb: 100
2015-02-12 09:42:34,991 INFO mapred.MapTask (MapTask.java:init(996)) - soft limit at 83886080
2015-02-12 09:42:34,992 INFO mapred.MapTask (MapTask.java:init(997)) - bufstart = 0; bufvoid = 104857600
2015-02-12 09:42:34,992 INFO mapred.MapTask (MapTask.java:init(998)) - kvstart = 26214396; length = 6553600
2015-02-12 09:42:34,993 INFO mapred.MapTask (MapTask.java:createSortingCollector(402)) - Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer
2015-02-12 09:42:34,993 INFO hadoop.FlowMapper (FlowMapper.java:configure(71)) - cascading version: 2.6.3
2015-02-12 09:42:34,993 INFO hadoop.FlowMapper (FlowMapper.java:configure(72)) - child jvm opts: -Xmx200m
2015-02-12 09:42:34,999 INFO hadoop.FlowMapper (FlowMapper.java:configure(87)) - sourcing from: Hfs["TextLine[[0:1]->[ALL]]"]["/Users/soberoi/src/compatibility/cascading.compatibility/horton/build/classes/test/data/apache.10.txt"]
2015-02-12 09:42:34,999 INFO hadoop.FlowMapper (FlowMapper.java:configure(90)) - sinking to: GroupBy(test)[by:[{1}:0]]
2015-02-12 09:42:35,002 INFO mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) -
2015-02-12 09:42:35,002 INFO mapred.MapTask (MapTask.java:flush(1457)) - Starting flush of map output
2015-02-12 09:42:35,002 INFO mapred.MapTask (MapTask.java:flush(1475)) - Spilling map output
2015-02-12 09:42:35,002 INFO mapred.MapTask (MapTask.java:flush(1476)) - bufstart = 0; bufend = 210; bufvoid = 104857600
2015-02-12 09:42:35,002 INFO mapred.MapTask (MapTask.java:flush(1478)) - kvstart = 26214396(104857584); kvend = 26214360(104857440); length = 37/6553600
2015-02-12 09:42:35,004 INFO mapred.MapTask (MapTask.java:sortAndSpill(1660)) - Finished spill 0
2015-02-12 09:42:35,006 INFO mapred.Task (Task.java:done(1001)) - Task:attempt_local1186336270_0015_m_000000_0 is done. And is in the process of committing
2015-02-12 09:42:35,007 INFO mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) - file:/Users/soberoi/src/compatibility/cascading.compatibility/horton/build/classes/test/data/apache.10.txt:0+1449
2015-02-12 09:42:35,007 INFO mapred.Task (Task.java:sendDone(1121)) - Task 'attempt_local1186336270_0015_m_000000_0' done.
2015-02-12 09:42:35,007 INFO mapred.LocalJobRunner (LocalJobRunner.java:run(249)) - Finishing task: attempt_local1186336270_0015_m_000000_0
2015-02-12 09:42:35,007 INFO mapred.LocalJobRunner (LocalJobRunner.java:runTasks(456)) - map task executor complete.
2015-02-12 09:42:35,008 INFO mapred.LocalJobRunner (LocalJobRunner.java:runTasks(448)) - Waiting for reduce tasks
2015-02-12 09:42:35,008 INFO mapred.LocalJobRunner (LocalJobRunner.java:run(302)) - Starting task: attempt_local1186336270_0015_r_000000_0
2015-02-12 09:42:35,009 INFO util.ProcfsBasedProcessTree (ProcfsBasedProcessTree.java:isAvailable(181)) - ProcfsBasedProcessTree currently is supported only on Linux.
2015-02-12 09:42:35,009 INFO mapred.Task (Task.java:initialize(587)) - Using ResourceCalculatorProcessTree : null
2015-02-12 09:42:35,009 INFO mapred.ReduceTask (ReduceTask.java:run(362)) - Using ShuffleConsumerPlugin: org.apache.hadoop.mapreduce.task.reduce.Shuffle@cb1e99b
2015-02-12 09:42:35,010 INFO reduce.MergeManagerImpl (MergeManagerImpl.java:<init>(196)) - MergerManager: memoryLimit=493250144, maxSingleShuffleLimit=123312536, mergeThreshold=325545120, ioSortFactor=10, memToMemMergeOutputsThreshold=10
2015-02-12 09:42:35,010 INFO reduce.EventFetcher (EventFetcher.java:run(61)) - attempt_local1186336270_0015_r_000000_0 Thread started: EventFetcher for fetching Map Completion Events
2015-02-12 09:42:35,011 INFO reduce.LocalFetcher (LocalFetcher.java:copyMapOutput(141)) - localfetcher#10 about to shuffle output of map attempt_local1186336270_0015_m_000000_0 decomp: 232 len: 236 to MEMORY
2015-02-12 09:42:35,012 INFO reduce.InMemoryMapOutput (InMemoryMapOutput.java:shuffle(100)) - Read 232 bytes from map-output for attempt_local1186336270_0015_m_000000_0
2015-02-12 09:42:35,012 INFO reduce.MergeManagerImpl (MergeManagerImpl.java:closeInMemoryFile(314)) - closeInMemoryFile -> map-output of size: 232, inMemoryMapOutputs.size() -> 1, commitMemory -> 0, usedMemory ->232
2015-02-12 09:42:35,012 INFO reduce.EventFetcher (EventFetcher.java:run(76)) - EventFetcher is interrupted.. Returning
2015-02-12 09:42:35,013 INFO mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) - 1 / 1 copied.
2015-02-12 09:42:35,013 INFO reduce.MergeManagerImpl (MergeManagerImpl.java:finalMerge(674)) - finalMerge called with 1 in-memory map-outputs and 0 on-disk map-outputs
2015-02-12 09:42:35,014 INFO mapred.Merger (Merger.java:merge(597)) - Merging 1 sorted segments
2015-02-12 09:42:35,015 INFO mapred.Merger (Merger.java:merge(696)) - Down to the last merge-pass, with 1 segments left of total size: 211 bytes
2015-02-12 09:42:35,016 INFO reduce.MergeManagerImpl (MergeManagerImpl.java:finalMerge(751)) - Merged 1 segments, 232 bytes to disk to satisfy reduce memory limit
2015-02-12 09:42:35,016 INFO reduce.MergeManagerImpl (MergeManagerImpl.java:finalMerge(781)) - Merging 1 files, 236 bytes from disk
2015-02-12 09:42:35,016 INFO reduce.MergeManagerImpl (MergeManagerImpl.java:finalMerge(796)) - Merging 0 segments, 0 bytes from memory into reduce
2015-02-12 09:42:35,016 INFO mapred.Merger (Merger.java:merge(597)) - Merging 1 sorted segments
2015-02-12 09:42:35,017 INFO mapred.Merger (Merger.java:merge(696)) - Down to the last merge-pass, with 1 segments left of total size: 211 bytes
2015-02-12 09:42:35,017 INFO mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) - 1 / 1 copied.
2015-02-12 09:42:35,018 INFO hadoop.FlowReducer (FlowReducer.java:configure(78)) - cascading version: 2.6.3
2015-02-12 09:42:35,018 INFO hadoop.FlowReducer (FlowReducer.java:configure(79)) - child jvm opts: -Xmx200m
2015-02-12 09:42:35,024 INFO hadoop.FlowReducer (FlowReducer.java:configure(97)) - sourcing from: GroupBy(test)[by:[{1}:0]]
2015-02-12 09:42:35,024 INFO hadoop.FlowReducer (FlowReducer.java:configure(100)) - sinking to: Hfs["TextLine[[0]->[ALL]]"]["/Users/soberoi/src/compatibility/cascading.compatibility/horton/build/test/output/hadoop2-mr1/basicpipesplatform/simple"]
2015-02-12 09:42:35,034 INFO mapred.Task (Task.java:done(1001)) - Task:attempt_local1186336270_0015_r_000000_0 is done. And is in the process of committing
2015-02-12 09:42:35,035 INFO mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) - 1 / 1 copied.
2015-02-12 09:42:35,035 INFO mapred.Task (Task.java:commit(1162)) - Task attempt_local1186336270_0015_r_000000_0 is allowed to commit now
2015-02-12 09:42:35,036 INFO output.FileOutputCommitter (FileOutputCommitter.java:commitTask(439)) - Saved output of task 'attempt_local1186336270_0015_r_000000_0' to file:/Users/soberoi/src/compatibility/cascading.compatibility/horton/build/test/output/hadoop2-mr1/basicpipesplatform/simple/_temporary/0/task_local1186336270_0015_r_000000
2015-02-12 09:42:35,037 INFO mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) - reduce > reduce
2015-02-12 09:42:35,038 INFO mapred.Task (Task.java:sendDone(1121)) - Task 'attempt_local1186336270_0015_r_000000_0' done.
2015-02-12 09:42:35,038 INFO mapred.LocalJobRunner (LocalJobRunner.java:run(325)) - Finishing task: attempt_local1186336270_0015_r_000000_0
2015-02-12 09:42:35,038 INFO mapred.LocalJobRunner (LocalJobRunner.java:runTasks(456)) - reduce task executor complete.
2015-02-12 09:42:35,052 INFO util.Hadoop18TapUtil (Hadoop18TapUtil.java:cleanTempPath(219)) - deleting temp path /Users/soberoi/src/compatibility/cascading.compatibility/horton/build/test/output/hadoop2-mr1/basicpipesplatform/simple/_temporary
2015-02-12 09:42:35,074 INFO mapred.FileInputFormat (FileInputFormat.java:listStatus(247)) - Total input paths to process : 1
2015-02-12 09:42:35,075 INFO cascading.PlatformTestCase (PlatformTestCase.java:tearDown(209)) - copying to local /Users/soberoi/src/compatibility/cascading.compatibility/horton/build/test/output//hadoop2-mr1/basicpipesplatform/simple