Class cascading.pipe.assembly.AssemblyHelpersPlatformTest

33

tests

0

failures

0

ignored

11.239s

duration

100%

successful

Tests

Test Duration Result
testAverageBy 0.228s passed
testAverageByNull 0.225s passed
testAverageMerge 0.539s passed
testCoerce 0.185s passed
testCoerceFields 0.177s passed
testCount 0.239s passed
testCountAll 0.239s passed
testCountCount 0.521s passed
testCountMerge 0.358s passed
testCountNullNotNull 0.222s passed
testDiscardNarrow 0.178s passed
testFirstBy 0.239s passed
testFirstByWithoutComparator 0.205s passed
testMaxBy 0.250s passed
testMaxByNullSafety 0.225s passed
testMaxByString 0.316s passed
testMinBy 0.272s passed
testMinByNullSafety 0.282s passed
testMinByString 0.360s passed
testParallelAggregates 0.239s passed
testParallelAggregatesMerge 2.077s passed
testParallelAggregatesPriorMerge 0.316s passed
testRenameAll 0.206s passed
testRenameNamed 0.173s passed
testRenameNarrow 0.155s passed
testRetainNarrow 0.179s passed
testSameSourceMerge 0.439s passed
testSameSourceMergeThreeWay 1.007s passed
testSumBy 0.245s passed
testSumByNulls 0.234s passed
testSumMerge 0.288s passed
testUnique 0.188s passed
testUniqueMerge 0.233s passed

Standard output

2015-02-12 16:12:43,740 INFO  platform.PlatformRunner (PlatformRunner.java:getPlatformClassesFromClasspath(187)) - found 1 test platforms from classpath
2015-02-12 16:12:43,773 INFO  platform.PlatformRunner (PlatformRunner.java:addPlatform(237)) - installing platform: hadoop2-mr1
2015-02-12 16:12:43,773 INFO  platform.PlatformRunner (PlatformRunner.java:addPlatform(238)) - running test: cascading.pipe.assembly.AssemblyHelpersPlatformTest
2015-02-12 16:12:43,795 INFO  hadoop2.Hadoop2MR1Platform (Hadoop2MR1Platform.java:setUp(105)) - not using cluster
2015-02-12 16:12:44,230 WARN  util.NativeCodeLoader (NativeCodeLoader.java:<clinit>(62)) - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
2015-02-12 16:12:44,570 INFO  util.HadoopUtil (HadoopUtil.java:findMainClass(336)) - using default application jar, may cause class not found exceptions on the cluster
2015-02-12 16:12:44,571 INFO  planner.HadoopPlanner (HadoopPlanner.java:initialize(225)) - using application jar: /Users/soberoi/.gradle/caches/modules-2/files-2.1/cascading/cascading-hadoop2-mr1/2.6.3/9ae016e2b8d246b9c1de9168e715d9f1b971aa4a/cascading-hadoop2-mr1-2.6.3.jar
2015-02-12 16:12:44,578 INFO  property.AppProps (AppProps.java:getAppID(169)) - using app.id: AE80D2D9424D4C76941AE218CDDD15C7
2015-02-12 16:12:44,662 INFO  Configuration.deprecation (Configuration.java:warnOnceIfDeprecated(1022)) - mapred.used.genericoptionsparser is deprecated. Instead, use mapreduce.client.genericoptionsparser.used
2015-02-12 16:12:44,692 INFO  Configuration.deprecation (Configuration.java:warnOnceIfDeprecated(1022)) - mapred.job.tracker is deprecated. Instead, use mapreduce.jobtracker.address
2015-02-12 16:12:44,695 INFO  Configuration.deprecation (Configuration.java:warnOnceIfDeprecated(1022)) - mapred.output.compress is deprecated. Instead, use mapreduce.output.fileoutputformat.compress
2015-02-12 16:12:44,804 INFO  util.Version (Version.java:printBanner(82)) - Concurrent, Inc - Cascading 2.6.3
2015-02-12 16:12:44,806 INFO  flow.Flow (BaseFlow.java:logInfo(1378)) - [name] starting
2015-02-12 16:12:44,806 INFO  flow.Flow (BaseFlow.java:logInfo(1378)) - [name]  source: Hfs["TextDelimited[['num', 'char']]"]["/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/classes/test/data/rhs.txt"]
2015-02-12 16:12:44,807 INFO  flow.Flow (BaseFlow.java:logInfo(1378)) - [name]  source: Hfs["TextDelimited[['num', 'char']]"]["/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/classes/test/data/lhs.txt"]
2015-02-12 16:12:44,807 INFO  flow.Flow (BaseFlow.java:logInfo(1378)) - [name]  sink: Hfs["TextDelimited[['char', 'sum', 'count', 'average']]"]["/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/test/output/hadoop2-mr1/assemblyhelpersplatform/multimerge+false"]
2015-02-12 16:12:44,808 INFO  flow.Flow (BaseFlow.java:logInfo(1378)) - [name]  parallel execution is enabled: false
2015-02-12 16:12:44,808 INFO  flow.Flow (BaseFlow.java:logInfo(1378)) - [name]  starting jobs: 1
2015-02-12 16:12:44,808 INFO  flow.Flow (BaseFlow.java:logInfo(1378)) - [name]  allocating threads: 1
2015-02-12 16:12:44,809 INFO  flow.FlowStep (BaseFlowStep.java:logInfo(834)) - [name] starting step: (1/1) ...platform/multimerge+false
2015-02-12 16:12:44,830 INFO  Configuration.deprecation (Configuration.java:warnOnceIfDeprecated(1022)) - session.id is deprecated. Instead, use dfs.metrics.session-id
2015-02-12 16:12:44,831 INFO  jvm.JvmMetrics (JvmMetrics.java:init(76)) - Initializing JVM Metrics with processName=JobTracker, sessionId=
2015-02-12 16:12:44,845 INFO  jvm.JvmMetrics (JvmMetrics.java:init(71)) - Cannot initialize JVM Metrics with processName=JobTracker, sessionId= - already initialized
2015-02-12 16:12:45,002 INFO  mapred.FileInputFormat (FileInputFormat.java:listStatus(247)) - Total input paths to process : 1
2015-02-12 16:12:45,012 INFO  mapred.FileInputFormat (FileInputFormat.java:listStatus(247)) - Total input paths to process : 1
2015-02-12 16:12:45,039 INFO  mapreduce.JobSubmitter (JobSubmitter.java:submitJobInternal(492)) - number of splits:2
2015-02-12 16:12:45,220 INFO  mapreduce.JobSubmitter (JobSubmitter.java:printTokens(575)) - Submitting tokens for job: job_local1069201491_0001
2015-02-12 16:12:45,275 WARN  conf.Configuration (Configuration.java:loadProperty(2510)) - file:/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/tmp/cascading/staging/soberoi1069201491/.staging/job_local1069201491_0001/job.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.retry.interval;  Ignoring.
2015-02-12 16:12:45,280 WARN  conf.Configuration (Configuration.java:loadProperty(2510)) - file:/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/tmp/cascading/staging/soberoi1069201491/.staging/job_local1069201491_0001/job.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.attempts;  Ignoring.
2015-02-12 16:12:45,357 WARN  conf.Configuration (Configuration.java:loadProperty(2510)) - file:/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/test/tmp/mapred/local/localRunner/soberoi/job_local1069201491_0001/job_local1069201491_0001.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.retry.interval;  Ignoring.
2015-02-12 16:12:45,359 WARN  conf.Configuration (Configuration.java:loadProperty(2510)) - file:/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/test/tmp/mapred/local/localRunner/soberoi/job_local1069201491_0001/job_local1069201491_0001.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.attempts;  Ignoring.
2015-02-12 16:12:45,362 INFO  mapreduce.Job (Job.java:submit(1299)) - The url to track the job: http://localhost:8080/
2015-02-12 16:12:45,363 INFO  mapred.LocalJobRunner (LocalJobRunner.java:createOutputCommitter(471)) - OutputCommitter set in config null
2015-02-12 16:12:45,364 INFO  flow.FlowStep (BaseFlowStep.java:logInfo(834)) - [name] submitted hadoop job: job_local1069201491_0001
2015-02-12 16:12:45,364 INFO  mapred.LocalJobRunner (LocalJobRunner.java:createOutputCommitter(489)) - OutputCommitter is org.apache.hadoop.mapred.FileOutputCommitter
2015-02-12 16:12:45,364 INFO  flow.FlowStep (BaseFlowStep.java:logInfo(834)) - [name] tracking url: http://localhost:8080/
2015-02-12 16:12:45,375 INFO  mapred.LocalJobRunner (LocalJobRunner.java:runTasks(448)) - Waiting for map tasks
2015-02-12 16:12:45,377 INFO  mapred.LocalJobRunner (LocalJobRunner.java:run(224)) - Starting task: attempt_local1069201491_0001_m_000000_0
2015-02-12 16:12:45,394 INFO  util.ProcfsBasedProcessTree (ProcfsBasedProcessTree.java:isAvailable(181)) - ProcfsBasedProcessTree currently is supported only on Linux.
2015-02-12 16:12:45,395 INFO  mapred.Task (Task.java:initialize(587)) -  Using ResourceCalculatorProcessTree : null
2015-02-12 16:12:45,399 INFO  io.MultiInputSplit (MultiInputSplit.java:readFields(161)) - current split input path: file:/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/classes/test/data/lhs.txt
2015-02-12 16:12:45,399 INFO  mapred.MapTask (MapTask.java:updateJobWithSplit(482)) - Processing split: cascading.tap.hadoop.io.MultiInputSplit@20d77214
2015-02-12 16:12:45,404 INFO  mapred.MapTask (MapTask.java:runOldMapper(436)) - numReduceTasks: 1
2015-02-12 16:12:45,438 INFO  mapred.MapTask (MapTask.java:setEquator(1202)) - (EQUATOR) 0 kvi 26214396(104857584)
2015-02-12 16:12:45,438 INFO  mapred.MapTask (MapTask.java:init(995)) - mapreduce.task.io.sort.mb: 100
2015-02-12 16:12:45,438 INFO  mapred.MapTask (MapTask.java:init(996)) - soft limit at 83886080
2015-02-12 16:12:45,438 INFO  mapred.MapTask (MapTask.java:init(997)) - bufstart = 0; bufvoid = 104857600
2015-02-12 16:12:45,438 INFO  mapred.MapTask (MapTask.java:init(998)) - kvstart = 26214396; length = 6553600
2015-02-12 16:12:45,453 INFO  mapred.MapTask (MapTask.java:createSortingCollector(402)) - Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer
2015-02-12 16:12:45,454 INFO  hadoop.FlowMapper (FlowMapper.java:configure(71)) - cascading version: 2.6.3
2015-02-12 16:12:45,454 INFO  hadoop.FlowMapper (FlowMapper.java:configure(72)) - child jvm opts: -Xmx200m
2015-02-12 16:12:45,484 INFO  Configuration.deprecation (Configuration.java:warnOnceIfDeprecated(1022)) - mapred.task.partition is deprecated. Instead, use mapreduce.task.partition
2015-02-12 16:12:45,488 INFO  hadoop.FlowMapper (FlowMapper.java:configure(87)) - sourcing from: Hfs["TextDelimited[['num', 'char']]"]["/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/classes/test/data/lhs.txt"]
2015-02-12 16:12:45,488 INFO  hadoop.FlowMapper (FlowMapper.java:configure(90)) - sinking to: GroupBy(name)[by: multi-lhs:[{1}:'char'] multi-rhs:[{1}:'char']]
2015-02-12 16:12:45,490 INFO  assembly.AggregateBy (AggregateBy.java:prepare(267)) - using threshold value: 2
2015-02-12 16:12:45,498 INFO  assembly.AggregateBy (AggregateBy.java:removeEldestEntry(327)) - flushed keys num times: 1, with threshold: 2
2015-02-12 16:12:45,498 INFO  assembly.AggregateBy (AggregateBy.java:removeEldestEntry(328)) - mem on flush (mb), free: 172, total: 310, max: 672
2015-02-12 16:12:45,498 INFO  assembly.AggregateBy (AggregateBy.java:removeEldestEntry(333)) - total mem is 46% of max mem, to better utilize unused memory consider increasing current LRU threshold with system property "cascading.aggregateby.threshold"
2015-02-12 16:12:45,499 INFO  assembly.AggregateBy (AggregateBy.java:removeEldestEntry(327)) - flushed keys num times: 3, with threshold: 2
2015-02-12 16:12:45,499 INFO  assembly.AggregateBy (AggregateBy.java:removeEldestEntry(328)) - mem on flush (mb), free: 172, total: 310, max: 672
2015-02-12 16:12:45,499 INFO  assembly.AggregateBy (AggregateBy.java:removeEldestEntry(333)) - total mem is 46% of max mem, to better utilize unused memory consider increasing current LRU threshold with system property "cascading.aggregateby.threshold"
2015-02-12 16:12:45,500 INFO  assembly.AggregateBy (AggregateBy.java:removeEldestEntry(327)) - flushed keys num times: 5, with threshold: 2
2015-02-12 16:12:45,500 INFO  assembly.AggregateBy (AggregateBy.java:removeEldestEntry(328)) - mem on flush (mb), free: 172, total: 310, max: 672
2015-02-12 16:12:45,500 INFO  assembly.AggregateBy (AggregateBy.java:removeEldestEntry(333)) - total mem is 46% of max mem, to better utilize unused memory consider increasing current LRU threshold with system property "cascading.aggregateby.threshold"
2015-02-12 16:12:45,501 INFO  assembly.AggregateBy (AggregateBy.java:removeEldestEntry(327)) - flushed keys num times: 7, with threshold: 2
2015-02-12 16:12:45,501 INFO  assembly.AggregateBy (AggregateBy.java:removeEldestEntry(328)) - mem on flush (mb), free: 172, total: 310, max: 672
2015-02-12 16:12:45,501 INFO  assembly.AggregateBy (AggregateBy.java:removeEldestEntry(333)) - total mem is 46% of max mem, to better utilize unused memory consider increasing current LRU threshold with system property "cascading.aggregateby.threshold"
2015-02-12 16:12:45,505 INFO  mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) - 
2015-02-12 16:12:45,505 INFO  mapred.MapTask (MapTask.java:flush(1457)) - Starting flush of map output
2015-02-12 16:12:45,505 INFO  mapred.MapTask (MapTask.java:flush(1475)) - Spilling map output
2015-02-12 16:12:45,505 INFO  mapred.MapTask (MapTask.java:flush(1476)) - bufstart = 0; bufend = 216; bufvoid = 104857600
2015-02-12 16:12:45,505 INFO  mapred.MapTask (MapTask.java:flush(1478)) - kvstart = 26214396(104857584); kvend = 26214364(104857456); length = 33/6553600
2015-02-12 16:12:45,511 INFO  mapred.MapTask (MapTask.java:sortAndSpill(1660)) - Finished spill 0
2015-02-12 16:12:45,514 INFO  mapred.Task (Task.java:done(1001)) - Task:attempt_local1069201491_0001_m_000000_0 is done. And is in the process of committing
2015-02-12 16:12:45,521 INFO  mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) - file:/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/classes/test/data/lhs.txt:0+52
2015-02-12 16:12:45,521 INFO  mapred.Task (Task.java:sendDone(1121)) - Task 'attempt_local1069201491_0001_m_000000_0' done.
2015-02-12 16:12:45,521 INFO  mapred.LocalJobRunner (LocalJobRunner.java:run(249)) - Finishing task: attempt_local1069201491_0001_m_000000_0
2015-02-12 16:12:45,521 INFO  mapred.LocalJobRunner (LocalJobRunner.java:run(224)) - Starting task: attempt_local1069201491_0001_m_000001_0
2015-02-12 16:12:45,524 INFO  util.ProcfsBasedProcessTree (ProcfsBasedProcessTree.java:isAvailable(181)) - ProcfsBasedProcessTree currently is supported only on Linux.
2015-02-12 16:12:45,524 INFO  mapred.Task (Task.java:initialize(587)) -  Using ResourceCalculatorProcessTree : null
2015-02-12 16:12:45,526 INFO  io.MultiInputSplit (MultiInputSplit.java:readFields(161)) - current split input path: file:/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/classes/test/data/rhs.txt
2015-02-12 16:12:45,526 INFO  mapred.MapTask (MapTask.java:updateJobWithSplit(482)) - Processing split: cascading.tap.hadoop.io.MultiInputSplit@64983afa
2015-02-12 16:12:45,528 INFO  mapred.MapTask (MapTask.java:runOldMapper(436)) - numReduceTasks: 1
2015-02-12 16:12:45,571 INFO  mapred.MapTask (MapTask.java:setEquator(1202)) - (EQUATOR) 0 kvi 26214396(104857584)
2015-02-12 16:12:45,571 INFO  mapred.MapTask (MapTask.java:init(995)) - mapreduce.task.io.sort.mb: 100
2015-02-12 16:12:45,571 INFO  mapred.MapTask (MapTask.java:init(996)) - soft limit at 83886080
2015-02-12 16:12:45,571 INFO  mapred.MapTask (MapTask.java:init(997)) - bufstart = 0; bufvoid = 104857600
2015-02-12 16:12:45,571 INFO  mapred.MapTask (MapTask.java:init(998)) - kvstart = 26214396; length = 6553600
2015-02-12 16:12:45,573 INFO  mapred.MapTask (MapTask.java:createSortingCollector(402)) - Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer
2015-02-12 16:12:45,574 INFO  hadoop.FlowMapper (FlowMapper.java:configure(71)) - cascading version: 2.6.3
2015-02-12 16:12:45,574 INFO  hadoop.FlowMapper (FlowMapper.java:configure(72)) - child jvm opts: -Xmx200m
2015-02-12 16:12:45,588 INFO  hadoop.FlowMapper (FlowMapper.java:configure(87)) - sourcing from: Hfs["TextDelimited[['num', 'char']]"]["/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/classes/test/data/rhs.txt"]
2015-02-12 16:12:45,588 INFO  hadoop.FlowMapper (FlowMapper.java:configure(90)) - sinking to: GroupBy(name)[by: multi-lhs:[{1}:'char'] multi-rhs:[{1}:'char']]
2015-02-12 16:12:45,588 INFO  assembly.AggregateBy (AggregateBy.java:prepare(267)) - using threshold value: 2
2015-02-12 16:12:45,663 INFO  assembly.AggregateBy (AggregateBy.java:removeEldestEntry(327)) - flushed keys num times: 1, with threshold: 2
2015-02-12 16:12:45,663 INFO  assembly.AggregateBy (AggregateBy.java:removeEldestEntry(328)) - mem on flush (mb), free: 198, total: 310, max: 672
2015-02-12 16:12:45,663 INFO  assembly.AggregateBy (AggregateBy.java:removeEldestEntry(333)) - total mem is 46% of max mem, to better utilize unused memory consider increasing current LRU threshold with system property "cascading.aggregateby.threshold"
2015-02-12 16:12:45,664 INFO  assembly.AggregateBy (AggregateBy.java:removeEldestEntry(327)) - flushed keys num times: 3, with threshold: 2
2015-02-12 16:12:45,664 INFO  assembly.AggregateBy (AggregateBy.java:removeEldestEntry(328)) - mem on flush (mb), free: 197, total: 310, max: 672
2015-02-12 16:12:45,664 INFO  assembly.AggregateBy (AggregateBy.java:removeEldestEntry(333)) - total mem is 46% of max mem, to better utilize unused memory consider increasing current LRU threshold with system property "cascading.aggregateby.threshold"
2015-02-12 16:12:45,665 INFO  assembly.AggregateBy (AggregateBy.java:removeEldestEntry(327)) - flushed keys num times: 5, with threshold: 2
2015-02-12 16:12:45,665 INFO  assembly.AggregateBy (AggregateBy.java:removeEldestEntry(328)) - mem on flush (mb), free: 197, total: 310, max: 672
2015-02-12 16:12:45,665 INFO  assembly.AggregateBy (AggregateBy.java:removeEldestEntry(333)) - total mem is 46% of max mem, to better utilize unused memory consider increasing current LRU threshold with system property "cascading.aggregateby.threshold"
2015-02-12 16:12:45,666 INFO  assembly.AggregateBy (AggregateBy.java:removeEldestEntry(327)) - flushed keys num times: 7, with threshold: 2
2015-02-12 16:12:45,666 INFO  assembly.AggregateBy (AggregateBy.java:removeEldestEntry(328)) - mem on flush (mb), free: 197, total: 310, max: 672
2015-02-12 16:12:45,666 INFO  assembly.AggregateBy (AggregateBy.java:removeEldestEntry(333)) - total mem is 46% of max mem, to better utilize unused memory consider increasing current LRU threshold with system property "cascading.aggregateby.threshold"
2015-02-12 16:12:45,667 INFO  mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) - 
2015-02-12 16:12:45,667 INFO  mapred.MapTask (MapTask.java:flush(1457)) - Starting flush of map output
2015-02-12 16:12:45,667 INFO  mapred.MapTask (MapTask.java:flush(1475)) - Spilling map output
2015-02-12 16:12:45,667 INFO  mapred.MapTask (MapTask.java:flush(1476)) - bufstart = 0; bufend = 216; bufvoid = 104857600
2015-02-12 16:12:45,667 INFO  mapred.MapTask (MapTask.java:flush(1478)) - kvstart = 26214396(104857584); kvend = 26214364(104857456); length = 33/6553600
2015-02-12 16:12:45,670 INFO  mapred.MapTask (MapTask.java:sortAndSpill(1660)) - Finished spill 0
2015-02-12 16:12:45,672 INFO  mapred.Task (Task.java:done(1001)) - Task:attempt_local1069201491_0001_m_000001_0 is done. And is in the process of committing
2015-02-12 16:12:45,674 INFO  mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) - file:/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/classes/test/data/rhs.txt:0+51
2015-02-12 16:12:45,674 INFO  mapred.Task (Task.java:sendDone(1121)) - Task 'attempt_local1069201491_0001_m_000001_0' done.
2015-02-12 16:12:45,674 INFO  mapred.LocalJobRunner (LocalJobRunner.java:run(249)) - Finishing task: attempt_local1069201491_0001_m_000001_0
2015-02-12 16:12:45,675 INFO  mapred.LocalJobRunner (LocalJobRunner.java:runTasks(456)) - map task executor complete.
2015-02-12 16:12:45,677 INFO  mapred.LocalJobRunner (LocalJobRunner.java:runTasks(448)) - Waiting for reduce tasks
2015-02-12 16:12:45,677 INFO  mapred.LocalJobRunner (LocalJobRunner.java:run(302)) - Starting task: attempt_local1069201491_0001_r_000000_0
2015-02-12 16:12:45,682 INFO  util.ProcfsBasedProcessTree (ProcfsBasedProcessTree.java:isAvailable(181)) - ProcfsBasedProcessTree currently is supported only on Linux.
2015-02-12 16:12:45,682 INFO  mapred.Task (Task.java:initialize(587)) -  Using ResourceCalculatorProcessTree : null
2015-02-12 16:12:45,684 INFO  mapred.ReduceTask (ReduceTask.java:run(362)) - Using ShuffleConsumerPlugin: org.apache.hadoop.mapreduce.task.reduce.Shuffle@1cd459c4
2015-02-12 16:12:45,691 INFO  reduce.MergeManagerImpl (MergeManagerImpl.java:<init>(195)) - MergerManager: memoryLimit=493250144, maxSingleShuffleLimit=123312536, mergeThreshold=325545120, ioSortFactor=10, memToMemMergeOutputsThreshold=10
2015-02-12 16:12:45,693 INFO  reduce.EventFetcher (EventFetcher.java:run(61)) - attempt_local1069201491_0001_r_000000_0 Thread started: EventFetcher for fetching Map Completion Events
2015-02-12 16:12:45,709 INFO  reduce.LocalFetcher (LocalFetcher.java:copyMapOutput(141)) - localfetcher#1 about to shuffle output of map attempt_local1069201491_0001_m_000001_0 decomp: 236 len: 240 to MEMORY
2015-02-12 16:12:45,711 INFO  reduce.InMemoryMapOutput (InMemoryMapOutput.java:shuffle(100)) - Read 236 bytes from map-output for attempt_local1069201491_0001_m_000001_0
2015-02-12 16:12:45,730 INFO  reduce.MergeManagerImpl (MergeManagerImpl.java:closeInMemoryFile(313)) - closeInMemoryFile -> map-output of size: 236, inMemoryMapOutputs.size() -> 1, commitMemory -> 0, usedMemory ->236
2015-02-12 16:12:45,732 INFO  reduce.LocalFetcher (LocalFetcher.java:copyMapOutput(141)) - localfetcher#1 about to shuffle output of map attempt_local1069201491_0001_m_000000_0 decomp: 236 len: 240 to MEMORY
2015-02-12 16:12:45,732 INFO  reduce.InMemoryMapOutput (InMemoryMapOutput.java:shuffle(100)) - Read 236 bytes from map-output for attempt_local1069201491_0001_m_000000_0
2015-02-12 16:12:45,733 INFO  reduce.MergeManagerImpl (MergeManagerImpl.java:closeInMemoryFile(313)) - closeInMemoryFile -> map-output of size: 236, inMemoryMapOutputs.size() -> 2, commitMemory -> 236, usedMemory ->472
2015-02-12 16:12:45,733 INFO  reduce.EventFetcher (EventFetcher.java:run(76)) - EventFetcher is interrupted.. Returning
2015-02-12 16:12:45,734 INFO  mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) - 2 / 2 copied.
2015-02-12 16:12:45,734 INFO  reduce.MergeManagerImpl (MergeManagerImpl.java:finalMerge(673)) - finalMerge called with 2 in-memory map-outputs and 0 on-disk map-outputs
2015-02-12 16:12:45,739 INFO  mapred.Merger (Merger.java:merge(606)) - Merging 2 sorted segments
2015-02-12 16:12:45,739 INFO  mapred.Merger (Merger.java:merge(705)) - Down to the last merge-pass, with 2 segments left of total size: 454 bytes
2015-02-12 16:12:45,741 INFO  reduce.MergeManagerImpl (MergeManagerImpl.java:finalMerge(750)) - Merged 2 segments, 472 bytes to disk to satisfy reduce memory limit
2015-02-12 16:12:45,742 INFO  reduce.MergeManagerImpl (MergeManagerImpl.java:finalMerge(780)) - Merging 1 files, 474 bytes from disk
2015-02-12 16:12:45,742 INFO  reduce.MergeManagerImpl (MergeManagerImpl.java:finalMerge(795)) - Merging 0 segments, 0 bytes from memory into reduce
2015-02-12 16:12:45,743 INFO  mapred.Merger (Merger.java:merge(606)) - Merging 1 sorted segments
2015-02-12 16:12:45,743 INFO  mapred.Merger (Merger.java:merge(705)) - Down to the last merge-pass, with 1 segments left of total size: 461 bytes
2015-02-12 16:12:45,744 INFO  mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) - 2 / 2 copied.
2015-02-12 16:12:45,746 INFO  hadoop.FlowReducer (FlowReducer.java:configure(78)) - cascading version: 2.6.3
2015-02-12 16:12:45,746 INFO  hadoop.FlowReducer (FlowReducer.java:configure(79)) - child jvm opts: -Xmx200m
2015-02-12 16:12:45,761 INFO  hadoop.FlowReducer (FlowReducer.java:configure(97)) - sourcing from: GroupBy(name)[by: multi-lhs:[{1}:'char'] multi-rhs:[{1}:'char']]
2015-02-12 16:12:45,761 INFO  hadoop.FlowReducer (FlowReducer.java:configure(100)) - sinking to: Hfs["TextDelimited[['char', 'sum', 'count', 'average']]"]["/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/test/output/hadoop2-mr1/assemblyhelpersplatform/multimerge+false"]
2015-02-12 16:12:45,781 INFO  mapred.Task (Task.java:done(1001)) - Task:attempt_local1069201491_0001_r_000000_0 is done. And is in the process of committing
2015-02-12 16:12:45,782 INFO  mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) - 2 / 2 copied.
2015-02-12 16:12:45,782 INFO  mapred.Task (Task.java:commit(1162)) - Task attempt_local1069201491_0001_r_000000_0 is allowed to commit now
2015-02-12 16:12:45,784 INFO  output.FileOutputCommitter (FileOutputCommitter.java:commitTask(439)) - Saved output of task 'attempt_local1069201491_0001_r_000000_0' to file:/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/test/output/hadoop2-mr1/assemblyhelpersplatform/multimerge+false/_temporary/0/task_local1069201491_0001_r_000000
2015-02-12 16:12:45,785 INFO  mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) - reduce > reduce
2015-02-12 16:12:45,785 INFO  mapred.Task (Task.java:sendDone(1121)) - Task 'attempt_local1069201491_0001_r_000000_0' done.
2015-02-12 16:12:45,785 INFO  mapred.LocalJobRunner (LocalJobRunner.java:run(325)) - Finishing task: attempt_local1069201491_0001_r_000000_0
2015-02-12 16:12:45,785 INFO  mapred.LocalJobRunner (LocalJobRunner.java:runTasks(456)) - reduce task executor complete.
2015-02-12 16:12:45,811 INFO  util.Hadoop18TapUtil (Hadoop18TapUtil.java:cleanTempPath(219)) - deleting temp path /Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/test/output/hadoop2-mr1/assemblyhelpersplatform/multimerge+false/_temporary
2015-02-12 16:12:45,837 INFO  mapred.FileInputFormat (FileInputFormat.java:listStatus(247)) - Total input paths to process : 1
2015-02-12 16:12:45,868 INFO  mapred.FileInputFormat (FileInputFormat.java:listStatus(247)) - Total input paths to process : 1
2015-02-12 16:12:45,870 INFO  cascading.PlatformTestCase (PlatformTestCase.java:tearDown(209)) - copying to local /Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/test/output//hadoop2-mr1/assemblyhelpersplatform/multimerge+false
2015-02-12 16:12:45,890 INFO  util.HadoopUtil (HadoopUtil.java:findMainClass(336)) - using default application jar, may cause class not found exceptions on the cluster
2015-02-12 16:12:45,891 INFO  planner.HadoopPlanner (HadoopPlanner.java:initialize(225)) - using application jar: /Users/soberoi/.gradle/caches/modules-2/files-2.1/cascading/cascading-hadoop2-mr1/2.6.3/9ae016e2b8d246b9c1de9168e715d9f1b971aa4a/cascading-hadoop2-mr1-2.6.3.jar
2015-02-12 16:12:45,899 INFO  Configuration.deprecation (Configuration.java:warnOnceIfDeprecated(1022)) - mapred.used.genericoptionsparser is deprecated. Instead, use mapreduce.client.genericoptionsparser.used
2015-02-12 16:12:45,902 INFO  Configuration.deprecation (Configuration.java:warnOnceIfDeprecated(1022)) - mapred.job.tracker is deprecated. Instead, use mapreduce.jobtracker.address
2015-02-12 16:12:45,903 INFO  Configuration.deprecation (Configuration.java:warnOnceIfDeprecated(1022)) - mapred.output.compress is deprecated. Instead, use mapreduce.output.fileoutputformat.compress
2015-02-12 16:12:45,911 INFO  flow.Flow (BaseFlow.java:logInfo(1378)) - [max] starting
2015-02-12 16:12:45,911 INFO  flow.Flow (BaseFlow.java:logInfo(1378)) - [max]  source: Hfs["TextDelimited[['num', 'char']]"]["/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/classes/test/data/lhs.txt"]
2015-02-12 16:12:45,912 INFO  flow.Flow (BaseFlow.java:logInfo(1378)) - [max]  sink: Hfs["TextDelimited[['num', 'min']]"]["/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/test/output/hadoop2-mr1/assemblyhelpersplatform/minbystring"]
2015-02-12 16:12:45,912 INFO  flow.Flow (BaseFlow.java:logInfo(1378)) - [max]  parallel execution is enabled: false
2015-02-12 16:12:45,912 INFO  flow.Flow (BaseFlow.java:logInfo(1378)) - [max]  starting jobs: 1
2015-02-12 16:12:45,912 INFO  flow.Flow (BaseFlow.java:logInfo(1378)) - [max]  allocating threads: 1
2015-02-12 16:12:45,912 INFO  flow.FlowStep (BaseFlowStep.java:logInfo(834)) - [max] starting step: (1/1) ...lpersplatform/minbystring
2015-02-12 16:12:45,913 INFO  jvm.JvmMetrics (JvmMetrics.java:init(71)) - Cannot initialize JVM Metrics with processName=JobTracker, sessionId= - already initialized
2015-02-12 16:12:45,915 INFO  jvm.JvmMetrics (JvmMetrics.java:init(71)) - Cannot initialize JVM Metrics with processName=JobTracker, sessionId= - already initialized
2015-02-12 16:12:45,937 INFO  mapred.FileInputFormat (FileInputFormat.java:listStatus(247)) - Total input paths to process : 1
2015-02-12 16:12:45,956 INFO  mapreduce.JobSubmitter (JobSubmitter.java:submitJobInternal(492)) - number of splits:1
2015-02-12 16:12:45,976 INFO  mapreduce.JobSubmitter (JobSubmitter.java:printTokens(575)) - Submitting tokens for job: job_local911131477_0002
2015-02-12 16:12:45,988 WARN  conf.Configuration (Configuration.java:loadProperty(2510)) - file:/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/tmp/cascading/staging/soberoi911131477/.staging/job_local911131477_0002/job.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.retry.interval;  Ignoring.
2015-02-12 16:12:45,991 WARN  conf.Configuration (Configuration.java:loadProperty(2510)) - file:/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/tmp/cascading/staging/soberoi911131477/.staging/job_local911131477_0002/job.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.attempts;  Ignoring.
2015-02-12 16:12:46,039 WARN  conf.Configuration (Configuration.java:loadProperty(2510)) - file:/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/test/tmp/mapred/local/localRunner/soberoi/job_local911131477_0002/job_local911131477_0002.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.retry.interval;  Ignoring.
2015-02-12 16:12:46,041 WARN  conf.Configuration (Configuration.java:loadProperty(2510)) - file:/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/test/tmp/mapred/local/localRunner/soberoi/job_local911131477_0002/job_local911131477_0002.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.attempts;  Ignoring.
2015-02-12 16:12:46,042 INFO  mapreduce.Job (Job.java:submit(1299)) - The url to track the job: http://localhost:8080/
2015-02-12 16:12:46,042 INFO  mapred.LocalJobRunner (LocalJobRunner.java:createOutputCommitter(471)) - OutputCommitter set in config null
2015-02-12 16:12:46,042 INFO  flow.FlowStep (BaseFlowStep.java:logInfo(834)) - [max] submitted hadoop job: job_local911131477_0002
2015-02-12 16:12:46,043 INFO  flow.FlowStep (BaseFlowStep.java:logInfo(834)) - [max] tracking url: http://localhost:8080/
2015-02-12 16:12:46,042 INFO  mapred.LocalJobRunner (LocalJobRunner.java:createOutputCommitter(489)) - OutputCommitter is org.apache.hadoop.mapred.FileOutputCommitter
2015-02-12 16:12:46,045 INFO  mapred.LocalJobRunner (LocalJobRunner.java:runTasks(448)) - Waiting for map tasks
2015-02-12 16:12:46,046 INFO  mapred.LocalJobRunner (LocalJobRunner.java:run(224)) - Starting task: attempt_local911131477_0002_m_000000_0
2015-02-12 16:12:46,047 INFO  util.ProcfsBasedProcessTree (ProcfsBasedProcessTree.java:isAvailable(181)) - ProcfsBasedProcessTree currently is supported only on Linux.
2015-02-12 16:12:46,047 INFO  mapred.Task (Task.java:initialize(587)) -  Using ResourceCalculatorProcessTree : null
2015-02-12 16:12:46,049 INFO  io.MultiInputSplit (MultiInputSplit.java:readFields(161)) - current split input path: file:/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/classes/test/data/lhs.txt
2015-02-12 16:12:46,049 INFO  mapred.MapTask (MapTask.java:updateJobWithSplit(482)) - Processing split: cascading.tap.hadoop.io.MultiInputSplit@3bf9fe08
2015-02-12 16:12:46,051 INFO  mapred.MapTask (MapTask.java:runOldMapper(436)) - numReduceTasks: 1
2015-02-12 16:12:46,104 INFO  mapred.MapTask (MapTask.java:setEquator(1202)) - (EQUATOR) 0 kvi 26214396(104857584)
2015-02-12 16:12:46,104 INFO  mapred.MapTask (MapTask.java:init(995)) - mapreduce.task.io.sort.mb: 100
2015-02-12 16:12:46,104 INFO  mapred.MapTask (MapTask.java:init(996)) - soft limit at 83886080
2015-02-12 16:12:46,104 INFO  mapred.MapTask (MapTask.java:init(997)) - bufstart = 0; bufvoid = 104857600
2015-02-12 16:12:46,104 INFO  mapred.MapTask (MapTask.java:init(998)) - kvstart = 26214396; length = 6553600
2015-02-12 16:12:46,106 INFO  mapred.MapTask (MapTask.java:createSortingCollector(402)) - Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer
2015-02-12 16:12:46,106 INFO  hadoop.FlowMapper (FlowMapper.java:configure(71)) - cascading version: 2.6.3
2015-02-12 16:12:46,106 INFO  hadoop.FlowMapper (FlowMapper.java:configure(72)) - child jvm opts: -Xmx200m
2015-02-12 16:12:46,113 INFO  hadoop.FlowMapper (FlowMapper.java:configure(87)) - sourcing from: Hfs["TextDelimited[['num', 'char']]"]["/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/classes/test/data/lhs.txt"]
2015-02-12 16:12:46,113 INFO  hadoop.FlowMapper (FlowMapper.java:configure(90)) - sinking to: GroupBy(max)[by:[{1}:'num']]
2015-02-12 16:12:46,113 INFO  assembly.AggregateBy (AggregateBy.java:prepare(267)) - using threshold value: 2
2015-02-12 16:12:46,115 INFO  assembly.AggregateBy (AggregateBy.java:removeEldestEntry(327)) - flushed keys num times: 1, with threshold: 2
2015-02-12 16:12:46,115 INFO  assembly.AggregateBy (AggregateBy.java:removeEldestEntry(328)) - mem on flush (mb), free: 129, total: 410, max: 672
2015-02-12 16:12:46,115 INFO  assembly.AggregateBy (AggregateBy.java:removeEldestEntry(333)) - total mem is 61% of max mem, to better utilize unused memory consider increasing current LRU threshold with system property "cascading.aggregateby.threshold"
2015-02-12 16:12:46,115 INFO  assembly.AggregateBy (AggregateBy.java:removeEldestEntry(327)) - flushed keys num times: 3, with threshold: 2
2015-02-12 16:12:46,115 INFO  assembly.AggregateBy (AggregateBy.java:removeEldestEntry(328)) - mem on flush (mb), free: 129, total: 410, max: 672
2015-02-12 16:12:46,115 INFO  assembly.AggregateBy (AggregateBy.java:removeEldestEntry(333)) - total mem is 61% of max mem, to better utilize unused memory consider increasing current LRU threshold with system property "cascading.aggregateby.threshold"
2015-02-12 16:12:46,116 INFO  mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) - 
2015-02-12 16:12:46,117 INFO  mapred.MapTask (MapTask.java:flush(1457)) - Starting flush of map output
2015-02-12 16:12:46,117 INFO  mapred.MapTask (MapTask.java:flush(1475)) - Spilling map output
2015-02-12 16:12:46,117 INFO  mapred.MapTask (MapTask.java:flush(1476)) - bufstart = 0; bufend = 75; bufvoid = 104857600
2015-02-12 16:12:46,117 INFO  mapred.MapTask (MapTask.java:flush(1478)) - kvstart = 26214396(104857584); kvend = 26214380(104857520); length = 17/6553600
2015-02-12 16:12:46,119 INFO  mapred.MapTask (MapTask.java:sortAndSpill(1660)) - Finished spill 0
2015-02-12 16:12:46,120 INFO  mapred.Task (Task.java:done(1001)) - Task:attempt_local911131477_0002_m_000000_0 is done. And is in the process of committing
2015-02-12 16:12:46,122 INFO  mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) - file:/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/classes/test/data/lhs.txt:0+52
2015-02-12 16:12:46,122 INFO  mapred.Task (Task.java:sendDone(1121)) - Task 'attempt_local911131477_0002_m_000000_0' done.
2015-02-12 16:12:46,122 INFO  mapred.LocalJobRunner (LocalJobRunner.java:run(249)) - Finishing task: attempt_local911131477_0002_m_000000_0
2015-02-12 16:12:46,122 INFO  mapred.LocalJobRunner (LocalJobRunner.java:runTasks(456)) - map task executor complete.
2015-02-12 16:12:46,123 INFO  mapred.LocalJobRunner (LocalJobRunner.java:runTasks(448)) - Waiting for reduce tasks
2015-02-12 16:12:46,123 INFO  mapred.LocalJobRunner (LocalJobRunner.java:run(302)) - Starting task: attempt_local911131477_0002_r_000000_0
2015-02-12 16:12:46,124 INFO  util.ProcfsBasedProcessTree (ProcfsBasedProcessTree.java:isAvailable(181)) - ProcfsBasedProcessTree currently is supported only on Linux.
2015-02-12 16:12:46,124 INFO  mapred.Task (Task.java:initialize(587)) -  Using ResourceCalculatorProcessTree : null
2015-02-12 16:12:46,124 INFO  mapred.ReduceTask (ReduceTask.java:run(362)) - Using ShuffleConsumerPlugin: org.apache.hadoop.mapreduce.task.reduce.Shuffle@3a2d2d6c
2015-02-12 16:12:46,125 INFO  reduce.MergeManagerImpl (MergeManagerImpl.java:<init>(195)) - MergerManager: memoryLimit=493250144, maxSingleShuffleLimit=123312536, mergeThreshold=325545120, ioSortFactor=10, memToMemMergeOutputsThreshold=10
2015-02-12 16:12:46,126 INFO  reduce.EventFetcher (EventFetcher.java:run(61)) - attempt_local911131477_0002_r_000000_0 Thread started: EventFetcher for fetching Map Completion Events
2015-02-12 16:12:46,126 INFO  reduce.LocalFetcher (LocalFetcher.java:copyMapOutput(141)) - localfetcher#2 about to shuffle output of map attempt_local911131477_0002_m_000000_0 decomp: 87 len: 91 to MEMORY
2015-02-12 16:12:46,127 INFO  reduce.InMemoryMapOutput (InMemoryMapOutput.java:shuffle(100)) - Read 87 bytes from map-output for attempt_local911131477_0002_m_000000_0
2015-02-12 16:12:46,127 INFO  reduce.MergeManagerImpl (MergeManagerImpl.java:closeInMemoryFile(313)) - closeInMemoryFile -> map-output of size: 87, inMemoryMapOutputs.size() -> 1, commitMemory -> 0, usedMemory ->87
2015-02-12 16:12:46,127 INFO  reduce.EventFetcher (EventFetcher.java:run(76)) - EventFetcher is interrupted.. Returning
2015-02-12 16:12:46,128 INFO  mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) - 1 / 1 copied.
2015-02-12 16:12:46,128 INFO  reduce.MergeManagerImpl (MergeManagerImpl.java:finalMerge(673)) - finalMerge called with 1 in-memory map-outputs and 0 on-disk map-outputs
2015-02-12 16:12:46,130 INFO  mapred.Merger (Merger.java:merge(606)) - Merging 1 sorted segments
2015-02-12 16:12:46,130 INFO  mapred.Merger (Merger.java:merge(705)) - Down to the last merge-pass, with 1 segments left of total size: 78 bytes
2015-02-12 16:12:46,132 INFO  reduce.MergeManagerImpl (MergeManagerImpl.java:finalMerge(750)) - Merged 1 segments, 87 bytes to disk to satisfy reduce memory limit
2015-02-12 16:12:46,132 INFO  reduce.MergeManagerImpl (MergeManagerImpl.java:finalMerge(780)) - Merging 1 files, 91 bytes from disk
2015-02-12 16:12:46,132 INFO  reduce.MergeManagerImpl (MergeManagerImpl.java:finalMerge(795)) - Merging 0 segments, 0 bytes from memory into reduce
2015-02-12 16:12:46,132 INFO  mapred.Merger (Merger.java:merge(606)) - Merging 1 sorted segments
2015-02-12 16:12:46,132 INFO  mapred.Merger (Merger.java:merge(705)) - Down to the last merge-pass, with 1 segments left of total size: 78 bytes
2015-02-12 16:12:46,133 INFO  mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) - 1 / 1 copied.
2015-02-12 16:12:46,133 INFO  hadoop.FlowReducer (FlowReducer.java:configure(78)) - cascading version: 2.6.3
2015-02-12 16:12:46,134 INFO  hadoop.FlowReducer (FlowReducer.java:configure(79)) - child jvm opts: -Xmx200m
2015-02-12 16:12:46,139 INFO  hadoop.FlowReducer (FlowReducer.java:configure(97)) - sourcing from: GroupBy(max)[by:[{1}:'num']]
2015-02-12 16:12:46,140 INFO  hadoop.FlowReducer (FlowReducer.java:configure(100)) - sinking to: Hfs["TextDelimited[['num', 'min']]"]["/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/test/output/hadoop2-mr1/assemblyhelpersplatform/minbystring"]
2015-02-12 16:12:46,149 INFO  mapred.Task (Task.java:done(1001)) - Task:attempt_local911131477_0002_r_000000_0 is done. And is in the process of committing
2015-02-12 16:12:46,150 INFO  mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) - 1 / 1 copied.
2015-02-12 16:12:46,150 INFO  mapred.Task (Task.java:commit(1162)) - Task attempt_local911131477_0002_r_000000_0 is allowed to commit now
2015-02-12 16:12:46,151 INFO  output.FileOutputCommitter (FileOutputCommitter.java:commitTask(439)) - Saved output of task 'attempt_local911131477_0002_r_000000_0' to file:/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/test/output/hadoop2-mr1/assemblyhelpersplatform/minbystring/_temporary/0/task_local911131477_0002_r_000000
2015-02-12 16:12:46,152 INFO  mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) - reduce > reduce
2015-02-12 16:12:46,152 INFO  mapred.Task (Task.java:sendDone(1121)) - Task 'attempt_local911131477_0002_r_000000_0' done.
2015-02-12 16:12:46,152 INFO  mapred.LocalJobRunner (LocalJobRunner.java:run(325)) - Finishing task: attempt_local911131477_0002_r_000000_0
2015-02-12 16:12:46,152 INFO  mapred.LocalJobRunner (LocalJobRunner.java:runTasks(456)) - reduce task executor complete.
2015-02-12 16:12:46,170 INFO  util.Hadoop18TapUtil (Hadoop18TapUtil.java:cleanTempPath(219)) - deleting temp path /Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/test/output/hadoop2-mr1/assemblyhelpersplatform/minbystring/_temporary
2015-02-12 16:12:46,203 INFO  mapred.FileInputFormat (FileInputFormat.java:listStatus(247)) - Total input paths to process : 1
2015-02-12 16:12:46,230 INFO  mapred.FileInputFormat (FileInputFormat.java:listStatus(247)) - Total input paths to process : 1
2015-02-12 16:12:46,231 INFO  cascading.PlatformTestCase (PlatformTestCase.java:tearDown(209)) - copying to local /Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/test/output//hadoop2-mr1/assemblyhelpersplatform/minbystring
2015-02-12 16:12:46,259 INFO  util.HadoopUtil (HadoopUtil.java:findMainClass(336)) - using default application jar, may cause class not found exceptions on the cluster
2015-02-12 16:12:46,260 INFO  planner.HadoopPlanner (HadoopPlanner.java:initialize(225)) - using application jar: /Users/soberoi/.gradle/caches/modules-2/files-2.1/cascading/cascading-hadoop2-mr1/2.6.3/9ae016e2b8d246b9c1de9168e715d9f1b971aa4a/cascading-hadoop2-mr1-2.6.3.jar
2015-02-12 16:12:46,275 INFO  Configuration.deprecation (Configuration.java:warnOnceIfDeprecated(1022)) - mapred.used.genericoptionsparser is deprecated. Instead, use mapreduce.client.genericoptionsparser.used
2015-02-12 16:12:46,282 INFO  Configuration.deprecation (Configuration.java:warnOnceIfDeprecated(1022)) - mapred.job.tracker is deprecated. Instead, use mapreduce.jobtracker.address
2015-02-12 16:12:46,283 INFO  Configuration.deprecation (Configuration.java:warnOnceIfDeprecated(1022)) - mapred.output.compress is deprecated. Instead, use mapreduce.output.fileoutputformat.compress
2015-02-12 16:12:46,294 INFO  flow.Flow (BaseFlow.java:logInfo(1378)) - [average-lhs+average-rhs] starting
2015-02-12 16:12:46,295 INFO  flow.Flow (BaseFlow.java:logInfo(1378)) - [average-lhs+average-rhs]  source: Hfs["TextDelimited[['num', 'char']]"]["/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/classes/test/data/rhs.txt"]
2015-02-12 16:12:46,295 INFO  flow.Flow (BaseFlow.java:logInfo(1378)) - [average-lhs+average-rhs]  source: Hfs["TextDelimited[['num', 'char']]"]["/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/classes/test/data/lhs.txt"]
2015-02-12 16:12:46,295 INFO  flow.Flow (BaseFlow.java:logInfo(1378)) - [average-lhs+average-rhs]  sink: Hfs["TextDelimited[['char', 'average']]"]["/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/test/output/hadoop2-mr1/assemblyhelpersplatform/mergeaverage"]
2015-02-12 16:12:46,295 INFO  flow.Flow (BaseFlow.java:logInfo(1378)) - [average-lhs+average-rhs]  parallel execution is enabled: false
2015-02-12 16:12:46,295 INFO  flow.Flow (BaseFlow.java:logInfo(1378)) - [average-lhs+average-rhs]  starting jobs: 1
2015-02-12 16:12:46,295 INFO  flow.Flow (BaseFlow.java:logInfo(1378)) - [average-lhs+average-rhs]  allocating threads: 1
2015-02-12 16:12:46,296 INFO  flow.FlowStep (BaseFlowStep.java:logInfo(834)) - [average-lhs+average-rhs] starting step: (1/1) ...persplatform/mergeaverage
2015-02-12 16:12:46,297 INFO  jvm.JvmMetrics (JvmMetrics.java:init(71)) - Cannot initialize JVM Metrics with processName=JobTracker, sessionId= - already initialized
2015-02-12 16:12:46,298 INFO  jvm.JvmMetrics (JvmMetrics.java:init(71)) - Cannot initialize JVM Metrics with processName=JobTracker, sessionId= - already initialized
2015-02-12 16:12:46,328 INFO  mapred.FileInputFormat (FileInputFormat.java:listStatus(247)) - Total input paths to process : 1
2015-02-12 16:12:46,329 INFO  mapred.FileInputFormat (FileInputFormat.java:listStatus(247)) - Total input paths to process : 1
2015-02-12 16:12:46,352 INFO  mapreduce.JobSubmitter (JobSubmitter.java:submitJobInternal(492)) - number of splits:2
2015-02-12 16:12:46,417 INFO  mapreduce.JobSubmitter (JobSubmitter.java:printTokens(575)) - Submitting tokens for job: job_local1056486563_0003
2015-02-12 16:12:46,436 WARN  conf.Configuration (Configuration.java:loadProperty(2510)) - file:/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/tmp/cascading/staging/soberoi1056486563/.staging/job_local1056486563_0003/job.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.retry.interval;  Ignoring.
2015-02-12 16:12:46,438 WARN  conf.Configuration (Configuration.java:loadProperty(2510)) - file:/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/tmp/cascading/staging/soberoi1056486563/.staging/job_local1056486563_0003/job.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.attempts;  Ignoring.
2015-02-12 16:12:46,535 WARN  conf.Configuration (Configuration.java:loadProperty(2510)) - file:/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/test/tmp/mapred/local/localRunner/soberoi/job_local1056486563_0003/job_local1056486563_0003.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.retry.interval;  Ignoring.
2015-02-12 16:12:46,536 WARN  conf.Configuration (Configuration.java:loadProperty(2510)) - file:/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/test/tmp/mapred/local/localRunner/soberoi/job_local1056486563_0003/job_local1056486563_0003.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.attempts;  Ignoring.
2015-02-12 16:12:46,537 INFO  mapreduce.Job (Job.java:submit(1299)) - The url to track the job: http://localhost:8080/
2015-02-12 16:12:46,537 INFO  mapred.LocalJobRunner (LocalJobRunner.java:createOutputCommitter(471)) - OutputCommitter set in config null
2015-02-12 16:12:46,537 INFO  flow.FlowStep (BaseFlowStep.java:logInfo(834)) - [average-lhs+average-rhs] submitted hadoop job: job_local1056486563_0003
2015-02-12 16:12:46,537 INFO  mapred.LocalJobRunner (LocalJobRunner.java:createOutputCommitter(489)) - OutputCommitter is org.apache.hadoop.mapred.FileOutputCommitter
2015-02-12 16:12:46,537 INFO  flow.FlowStep (BaseFlowStep.java:logInfo(834)) - [average-lhs+average-rhs] tracking url: http://localhost:8080/
2015-02-12 16:12:46,539 INFO  mapred.LocalJobRunner (LocalJobRunner.java:runTasks(448)) - Waiting for map tasks
2015-02-12 16:12:46,539 INFO  mapred.LocalJobRunner (LocalJobRunner.java:run(224)) - Starting task: attempt_local1056486563_0003_m_000000_0
2015-02-12 16:12:46,541 INFO  util.ProcfsBasedProcessTree (ProcfsBasedProcessTree.java:isAvailable(181)) - ProcfsBasedProcessTree currently is supported only on Linux.
2015-02-12 16:12:46,541 INFO  mapred.Task (Task.java:initialize(587)) -  Using ResourceCalculatorProcessTree : null
2015-02-12 16:12:46,542 INFO  io.MultiInputSplit (MultiInputSplit.java:readFields(161)) - current split input path: file:/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/classes/test/data/lhs.txt
2015-02-12 16:12:46,542 INFO  mapred.MapTask (MapTask.java:updateJobWithSplit(482)) - Processing split: cascading.tap.hadoop.io.MultiInputSplit@295d5fc0
2015-02-12 16:12:46,543 INFO  mapred.MapTask (MapTask.java:runOldMapper(436)) - numReduceTasks: 1
2015-02-12 16:12:46,555 INFO  mapred.MapTask (MapTask.java:setEquator(1202)) - (EQUATOR) 0 kvi 26214396(104857584)
2015-02-12 16:12:46,555 INFO  mapred.MapTask (MapTask.java:init(995)) - mapreduce.task.io.sort.mb: 100
2015-02-12 16:12:46,555 INFO  mapred.MapTask (MapTask.java:init(996)) - soft limit at 83886080
2015-02-12 16:12:46,555 INFO  mapred.MapTask (MapTask.java:init(997)) - bufstart = 0; bufvoid = 104857600
2015-02-12 16:12:46,555 INFO  mapred.MapTask (MapTask.java:init(998)) - kvstart = 26214396; length = 6553600
2015-02-12 16:12:46,557 INFO  mapred.MapTask (MapTask.java:createSortingCollector(402)) - Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer
2015-02-12 16:12:46,557 INFO  hadoop.FlowMapper (FlowMapper.java:configure(71)) - cascading version: 2.6.3
2015-02-12 16:12:46,557 INFO  hadoop.FlowMapper (FlowMapper.java:configure(72)) - child jvm opts: -Xmx200m
2015-02-12 16:12:46,567 INFO  hadoop.FlowMapper (FlowMapper.java:configure(87)) - sourcing from: Hfs["TextDelimited[['num', 'char']]"]["/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/classes/test/data/lhs.txt"]
2015-02-12 16:12:46,567 INFO  hadoop.FlowMapper (FlowMapper.java:configure(90)) - sinking to: GroupBy(average-lhs+average-rhs)[by: average-lhs:[{1}:'char'] average-rhs:[{1}:'char']]
2015-02-12 16:12:46,567 INFO  assembly.AggregateBy (AggregateBy.java:prepare(267)) - using threshold value: 2
2015-02-12 16:12:46,569 INFO  assembly.AggregateBy (AggregateBy.java:removeEldestEntry(327)) - flushed keys num times: 1, with threshold: 2
2015-02-12 16:12:46,569 INFO  assembly.AggregateBy (AggregateBy.java:removeEldestEntry(328)) - mem on flush (mb), free: 193, total: 512, max: 672
2015-02-12 16:12:46,569 INFO  assembly.AggregateBy (AggregateBy.java:removeEldestEntry(333)) - total mem is 76% of max mem, to better utilize unused memory consider increasing current LRU threshold with system property "cascading.aggregateby.threshold"
2015-02-12 16:12:46,569 INFO  assembly.AggregateBy (AggregateBy.java:removeEldestEntry(327)) - flushed keys num times: 3, with threshold: 2
2015-02-12 16:12:46,570 INFO  assembly.AggregateBy (AggregateBy.java:removeEldestEntry(328)) - mem on flush (mb), free: 193, total: 512, max: 672
2015-02-12 16:12:46,570 INFO  assembly.AggregateBy (AggregateBy.java:removeEldestEntry(333)) - total mem is 76% of max mem, to better utilize unused memory consider increasing current LRU threshold with system property "cascading.aggregateby.threshold"
2015-02-12 16:12:46,570 INFO  assembly.AggregateBy (AggregateBy.java:removeEldestEntry(327)) - flushed keys num times: 5, with threshold: 2
2015-02-12 16:12:46,570 INFO  assembly.AggregateBy (AggregateBy.java:removeEldestEntry(328)) - mem on flush (mb), free: 193, total: 512, max: 672
2015-02-12 16:12:46,570 INFO  assembly.AggregateBy (AggregateBy.java:removeEldestEntry(333)) - total mem is 76% of max mem, to better utilize unused memory consider increasing current LRU threshold with system property "cascading.aggregateby.threshold"
2015-02-12 16:12:46,571 INFO  assembly.AggregateBy (AggregateBy.java:removeEldestEntry(327)) - flushed keys num times: 7, with threshold: 2
2015-02-12 16:12:46,571 INFO  assembly.AggregateBy (AggregateBy.java:removeEldestEntry(328)) - mem on flush (mb), free: 193, total: 512, max: 672
2015-02-12 16:12:46,571 INFO  assembly.AggregateBy (AggregateBy.java:removeEldestEntry(333)) - total mem is 76% of max mem, to better utilize unused memory consider increasing current LRU threshold with system property "cascading.aggregateby.threshold"
2015-02-12 16:12:46,572 INFO  mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) - 
2015-02-12 16:12:46,572 INFO  mapred.MapTask (MapTask.java:flush(1457)) - Starting flush of map output
2015-02-12 16:12:46,573 INFO  mapred.MapTask (MapTask.java:flush(1475)) - Spilling map output
2015-02-12 16:12:46,573 INFO  mapred.MapTask (MapTask.java:flush(1476)) - bufstart = 0; bufend = 180; bufvoid = 104857600
2015-02-12 16:12:46,573 INFO  mapred.MapTask (MapTask.java:flush(1478)) - kvstart = 26214396(104857584); kvend = 26214364(104857456); length = 33/6553600
2015-02-12 16:12:46,576 INFO  mapred.MapTask (MapTask.java:sortAndSpill(1660)) - Finished spill 0
2015-02-12 16:12:46,578 INFO  mapred.Task (Task.java:done(1001)) - Task:attempt_local1056486563_0003_m_000000_0 is done. And is in the process of committing
2015-02-12 16:12:46,580 INFO  mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) - file:/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/classes/test/data/lhs.txt:0+52
2015-02-12 16:12:46,580 INFO  mapred.Task (Task.java:sendDone(1121)) - Task 'attempt_local1056486563_0003_m_000000_0' done.
2015-02-12 16:12:46,580 INFO  mapred.LocalJobRunner (LocalJobRunner.java:run(249)) - Finishing task: attempt_local1056486563_0003_m_000000_0
2015-02-12 16:12:46,580 INFO  mapred.LocalJobRunner (LocalJobRunner.java:run(224)) - Starting task: attempt_local1056486563_0003_m_000001_0
2015-02-12 16:12:46,582 INFO  util.ProcfsBasedProcessTree (ProcfsBasedProcessTree.java:isAvailable(181)) - ProcfsBasedProcessTree currently is supported only on Linux.
2015-02-12 16:12:46,582 INFO  mapred.Task (Task.java:initialize(587)) -  Using ResourceCalculatorProcessTree : null
2015-02-12 16:12:46,583 INFO  io.MultiInputSplit (MultiInputSplit.java:readFields(161)) - current split input path: file:/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/classes/test/data/rhs.txt
2015-02-12 16:12:46,584 INFO  mapred.MapTask (MapTask.java:updateJobWithSplit(482)) - Processing split: cascading.tap.hadoop.io.MultiInputSplit@43cefb4d
2015-02-12 16:12:46,585 INFO  mapred.MapTask (MapTask.java:runOldMapper(436)) - numReduceTasks: 1
2015-02-12 16:12:46,623 INFO  mapred.MapTask (MapTask.java:setEquator(1202)) - (EQUATOR) 0 kvi 26214396(104857584)
2015-02-12 16:12:46,623 INFO  mapred.MapTask (MapTask.java:init(995)) - mapreduce.task.io.sort.mb: 100
2015-02-12 16:12:46,624 INFO  mapred.MapTask (MapTask.java:init(996)) - soft limit at 83886080
2015-02-12 16:12:46,624 INFO  mapred.MapTask (MapTask.java:init(997)) - bufstart = 0; bufvoid = 104857600
2015-02-12 16:12:46,624 INFO  mapred.MapTask (MapTask.java:init(998)) - kvstart = 26214396; length = 6553600
2015-02-12 16:12:46,625 INFO  mapred.MapTask (MapTask.java:createSortingCollector(402)) - Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer
2015-02-12 16:12:46,625 INFO  hadoop.FlowMapper (FlowMapper.java:configure(71)) - cascading version: 2.6.3
2015-02-12 16:12:46,625 INFO  hadoop.FlowMapper (FlowMapper.java:configure(72)) - child jvm opts: -Xmx200m
2015-02-12 16:12:46,635 INFO  hadoop.FlowMapper (FlowMapper.java:configure(87)) - sourcing from: Hfs["TextDelimited[['num', 'char']]"]["/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/classes/test/data/rhs.txt"]
2015-02-12 16:12:46,635 INFO  hadoop.FlowMapper (FlowMapper.java:configure(90)) - sinking to: GroupBy(average-lhs+average-rhs)[by: average-lhs:[{1}:'char'] average-rhs:[{1}:'char']]
2015-02-12 16:12:46,635 INFO  assembly.AggregateBy (AggregateBy.java:prepare(267)) - using threshold value: 2
2015-02-12 16:12:46,638 INFO  assembly.AggregateBy (AggregateBy.java:removeEldestEntry(327)) - flushed keys num times: 1, with threshold: 2
2015-02-12 16:12:46,638 INFO  assembly.AggregateBy (AggregateBy.java:removeEldestEntry(328)) - mem on flush (mb), free: 91, total: 512, max: 672
2015-02-12 16:12:46,638 INFO  assembly.AggregateBy (AggregateBy.java:removeEldestEntry(333)) - total mem is 76% of max mem, to better utilize unused memory consider increasing current LRU threshold with system property "cascading.aggregateby.threshold"
2015-02-12 16:12:46,639 INFO  assembly.AggregateBy (AggregateBy.java:removeEldestEntry(327)) - flushed keys num times: 3, with threshold: 2
2015-02-12 16:12:46,639 INFO  assembly.AggregateBy (AggregateBy.java:removeEldestEntry(328)) - mem on flush (mb), free: 91, total: 512, max: 672
2015-02-12 16:12:46,639 INFO  assembly.AggregateBy (AggregateBy.java:removeEldestEntry(333)) - total mem is 76% of max mem, to better utilize unused memory consider increasing current LRU threshold with system property "cascading.aggregateby.threshold"
2015-02-12 16:12:46,640 INFO  assembly.AggregateBy (AggregateBy.java:removeEldestEntry(327)) - flushed keys num times: 5, with threshold: 2
2015-02-12 16:12:46,640 INFO  assembly.AggregateBy (AggregateBy.java:removeEldestEntry(328)) - mem on flush (mb), free: 91, total: 512, max: 672
2015-02-12 16:12:46,640 INFO  assembly.AggregateBy (AggregateBy.java:removeEldestEntry(333)) - total mem is 76% of max mem, to better utilize unused memory consider increasing current LRU threshold with system property "cascading.aggregateby.threshold"
2015-02-12 16:12:46,641 INFO  assembly.AggregateBy (AggregateBy.java:removeEldestEntry(327)) - flushed keys num times: 7, with threshold: 2
2015-02-12 16:12:46,641 INFO  assembly.AggregateBy (AggregateBy.java:removeEldestEntry(328)) - mem on flush (mb), free: 91, total: 512, max: 672
2015-02-12 16:12:46,641 INFO  assembly.AggregateBy (AggregateBy.java:removeEldestEntry(333)) - total mem is 76% of max mem, to better utilize unused memory consider increasing current LRU threshold with system property "cascading.aggregateby.threshold"
2015-02-12 16:12:46,642 INFO  mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) - 
2015-02-12 16:12:46,642 INFO  mapred.MapTask (MapTask.java:flush(1457)) - Starting flush of map output
2015-02-12 16:12:46,642 INFO  mapred.MapTask (MapTask.java:flush(1475)) - Spilling map output
2015-02-12 16:12:46,642 INFO  mapred.MapTask (MapTask.java:flush(1476)) - bufstart = 0; bufend = 180; bufvoid = 104857600
2015-02-12 16:12:46,642 INFO  mapred.MapTask (MapTask.java:flush(1478)) - kvstart = 26214396(104857584); kvend = 26214364(104857456); length = 33/6553600
2015-02-12 16:12:46,646 INFO  mapred.MapTask (MapTask.java:sortAndSpill(1660)) - Finished spill 0
2015-02-12 16:12:46,648 INFO  mapred.Task (Task.java:done(1001)) - Task:attempt_local1056486563_0003_m_000001_0 is done. And is in the process of committing
2015-02-12 16:12:46,650 INFO  mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) - file:/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/classes/test/data/rhs.txt:0+51
2015-02-12 16:12:46,650 INFO  mapred.Task (Task.java:sendDone(1121)) - Task 'attempt_local1056486563_0003_m_000001_0' done.
2015-02-12 16:12:46,650 INFO  mapred.LocalJobRunner (LocalJobRunner.java:run(249)) - Finishing task: attempt_local1056486563_0003_m_000001_0
2015-02-12 16:12:46,650 INFO  mapred.LocalJobRunner (LocalJobRunner.java:runTasks(456)) - map task executor complete.
2015-02-12 16:12:46,650 INFO  mapred.LocalJobRunner (LocalJobRunner.java:runTasks(448)) - Waiting for reduce tasks
2015-02-12 16:12:46,651 INFO  mapred.LocalJobRunner (LocalJobRunner.java:run(302)) - Starting task: attempt_local1056486563_0003_r_000000_0
2015-02-12 16:12:46,652 INFO  util.ProcfsBasedProcessTree (ProcfsBasedProcessTree.java:isAvailable(181)) - ProcfsBasedProcessTree currently is supported only on Linux.
2015-02-12 16:12:46,652 INFO  mapred.Task (Task.java:initialize(587)) -  Using ResourceCalculatorProcessTree : null
2015-02-12 16:12:46,652 INFO  mapred.ReduceTask (ReduceTask.java:run(362)) - Using ShuffleConsumerPlugin: org.apache.hadoop.mapreduce.task.reduce.Shuffle@5c4e4be3
2015-02-12 16:12:46,653 INFO  reduce.MergeManagerImpl (MergeManagerImpl.java:<init>(195)) - MergerManager: memoryLimit=493250144, maxSingleShuffleLimit=123312536, mergeThreshold=325545120, ioSortFactor=10, memToMemMergeOutputsThreshold=10
2015-02-12 16:12:46,653 INFO  reduce.EventFetcher (EventFetcher.java:run(61)) - attempt_local1056486563_0003_r_000000_0 Thread started: EventFetcher for fetching Map Completion Events
2015-02-12 16:12:46,655 INFO  reduce.LocalFetcher (LocalFetcher.java:copyMapOutput(141)) - localfetcher#3 about to shuffle output of map attempt_local1056486563_0003_m_000001_0 decomp: 200 len: 204 to MEMORY
2015-02-12 16:12:46,655 INFO  reduce.InMemoryMapOutput (InMemoryMapOutput.java:shuffle(100)) - Read 200 bytes from map-output for attempt_local1056486563_0003_m_000001_0
2015-02-12 16:12:46,655 INFO  reduce.MergeManagerImpl (MergeManagerImpl.java:closeInMemoryFile(313)) - closeInMemoryFile -> map-output of size: 200, inMemoryMapOutputs.size() -> 1, commitMemory -> 0, usedMemory ->200
2015-02-12 16:12:46,657 INFO  reduce.LocalFetcher (LocalFetcher.java:copyMapOutput(141)) - localfetcher#3 about to shuffle output of map attempt_local1056486563_0003_m_000000_0 decomp: 200 len: 204 to MEMORY
2015-02-12 16:12:46,658 INFO  reduce.InMemoryMapOutput (InMemoryMapOutput.java:shuffle(100)) - Read 200 bytes from map-output for attempt_local1056486563_0003_m_000000_0
2015-02-12 16:12:46,658 INFO  reduce.MergeManagerImpl (MergeManagerImpl.java:closeInMemoryFile(313)) - closeInMemoryFile -> map-output of size: 200, inMemoryMapOutputs.size() -> 2, commitMemory -> 200, usedMemory ->400
2015-02-12 16:12:46,658 INFO  reduce.EventFetcher (EventFetcher.java:run(76)) - EventFetcher is interrupted.. Returning
2015-02-12 16:12:46,659 INFO  mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) - 2 / 2 copied.
2015-02-12 16:12:46,659 INFO  reduce.MergeManagerImpl (MergeManagerImpl.java:finalMerge(673)) - finalMerge called with 2 in-memory map-outputs and 0 on-disk map-outputs
2015-02-12 16:12:46,661 INFO  mapred.Merger (Merger.java:merge(606)) - Merging 2 sorted segments
2015-02-12 16:12:46,661 INFO  mapred.Merger (Merger.java:merge(705)) - Down to the last merge-pass, with 2 segments left of total size: 382 bytes
2015-02-12 16:12:46,662 INFO  reduce.MergeManagerImpl (MergeManagerImpl.java:finalMerge(750)) - Merged 2 segments, 400 bytes to disk to satisfy reduce memory limit
2015-02-12 16:12:46,663 INFO  reduce.MergeManagerImpl (MergeManagerImpl.java:finalMerge(780)) - Merging 1 files, 402 bytes from disk
2015-02-12 16:12:46,663 INFO  reduce.MergeManagerImpl (MergeManagerImpl.java:finalMerge(795)) - Merging 0 segments, 0 bytes from memory into reduce
2015-02-12 16:12:46,663 INFO  mapred.Merger (Merger.java:merge(606)) - Merging 1 sorted segments
2015-02-12 16:12:46,663 INFO  mapred.Merger (Merger.java:merge(705)) - Down to the last merge-pass, with 1 segments left of total size: 389 bytes
2015-02-12 16:12:46,664 INFO  mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) - 2 / 2 copied.
2015-02-12 16:12:46,664 INFO  hadoop.FlowReducer (FlowReducer.java:configure(78)) - cascading version: 2.6.3
2015-02-12 16:12:46,664 INFO  hadoop.FlowReducer (FlowReducer.java:configure(79)) - child jvm opts: -Xmx200m
2015-02-12 16:12:46,672 INFO  hadoop.FlowReducer (FlowReducer.java:configure(97)) - sourcing from: GroupBy(average-lhs+average-rhs)[by: average-lhs:[{1}:'char'] average-rhs:[{1}:'char']]
2015-02-12 16:12:46,672 INFO  hadoop.FlowReducer (FlowReducer.java:configure(100)) - sinking to: Hfs["TextDelimited[['char', 'average']]"]["/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/test/output/hadoop2-mr1/assemblyhelpersplatform/mergeaverage"]
2015-02-12 16:12:46,681 INFO  mapred.Task (Task.java:done(1001)) - Task:attempt_local1056486563_0003_r_000000_0 is done. And is in the process of committing
2015-02-12 16:12:46,682 INFO  mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) - 2 / 2 copied.
2015-02-12 16:12:46,683 INFO  mapred.Task (Task.java:commit(1162)) - Task attempt_local1056486563_0003_r_000000_0 is allowed to commit now
2015-02-12 16:12:46,683 INFO  output.FileOutputCommitter (FileOutputCommitter.java:commitTask(439)) - Saved output of task 'attempt_local1056486563_0003_r_000000_0' to file:/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/test/output/hadoop2-mr1/assemblyhelpersplatform/mergeaverage/_temporary/0/task_local1056486563_0003_r_000000
2015-02-12 16:12:46,684 INFO  mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) - reduce > reduce
2015-02-12 16:12:46,684 INFO  mapred.Task (Task.java:sendDone(1121)) - Task 'attempt_local1056486563_0003_r_000000_0' done.
2015-02-12 16:12:46,684 INFO  mapred.LocalJobRunner (LocalJobRunner.java:run(325)) - Finishing task: attempt_local1056486563_0003_r_000000_0
2015-02-12 16:12:46,685 INFO  mapred.LocalJobRunner (LocalJobRunner.java:runTasks(456)) - reduce task executor complete.
2015-02-12 16:12:46,705 INFO  util.Hadoop18TapUtil (Hadoop18TapUtil.java:cleanTempPath(219)) - deleting temp path /Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/test/output/hadoop2-mr1/assemblyhelpersplatform/mergeaverage/_temporary
2015-02-12 16:12:46,746 INFO  mapred.FileInputFormat (FileInputFormat.java:listStatus(247)) - Total input paths to process : 1
2015-02-12 16:12:46,770 INFO  mapred.FileInputFormat (FileInputFormat.java:listStatus(247)) - Total input paths to process : 1
2015-02-12 16:12:46,771 INFO  cascading.PlatformTestCase (PlatformTestCase.java:tearDown(209)) - copying to local /Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/test/output//hadoop2-mr1/assemblyhelpersplatform/mergeaverage
2015-02-12 16:12:46,786 INFO  util.HadoopUtil (HadoopUtil.java:findMainClass(336)) - using default application jar, may cause class not found exceptions on the cluster
2015-02-12 16:12:46,787 INFO  planner.HadoopPlanner (HadoopPlanner.java:initialize(225)) - using application jar: /Users/soberoi/.gradle/caches/modules-2/files-2.1/cascading/cascading-hadoop2-mr1/2.6.3/9ae016e2b8d246b9c1de9168e715d9f1b971aa4a/cascading-hadoop2-mr1-2.6.3.jar
2015-02-12 16:12:46,792 INFO  Configuration.deprecation (Configuration.java:warnOnceIfDeprecated(1022)) - mapred.used.genericoptionsparser is deprecated. Instead, use mapreduce.client.genericoptionsparser.used
2015-02-12 16:12:46,793 INFO  Configuration.deprecation (Configuration.java:warnOnceIfDeprecated(1022)) - mapred.job.tracker is deprecated. Instead, use mapreduce.jobtracker.address
2015-02-12 16:12:46,794 INFO  Configuration.deprecation (Configuration.java:warnOnceIfDeprecated(1022)) - mapred.output.compress is deprecated. Instead, use mapreduce.output.fileoutputformat.compress
2015-02-12 16:12:46,802 INFO  flow.Flow (BaseFlow.java:logInfo(1378)) - [shape] starting
2015-02-12 16:12:46,802 INFO  flow.Flow (BaseFlow.java:logInfo(1378)) - [shape]  source: Hfs["TextLine[['offset', 'line']->[ALL]]"]["/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/classes/test/data/lower.txt"]
2015-02-12 16:12:46,802 INFO  flow.Flow (BaseFlow.java:logInfo(1378)) - [shape]  sink: Hfs["TextLine[['line']->['item', 'element']]"]["/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/test/output/hadoop2-mr1/assemblyhelpersplatform/renameall"]
2015-02-12 16:12:46,803 INFO  flow.Flow (BaseFlow.java:logInfo(1378)) - [shape]  parallel execution is enabled: false
2015-02-12 16:12:46,803 INFO  flow.Flow (BaseFlow.java:logInfo(1378)) - [shape]  starting jobs: 1
2015-02-12 16:12:46,803 INFO  flow.Flow (BaseFlow.java:logInfo(1378)) - [shape]  allocating threads: 1
2015-02-12 16:12:46,803 INFO  flow.FlowStep (BaseFlowStep.java:logInfo(834)) - [shape] starting step: (1/1) ...helpersplatform/renameall
2015-02-12 16:12:46,804 INFO  jvm.JvmMetrics (JvmMetrics.java:init(71)) - Cannot initialize JVM Metrics with processName=JobTracker, sessionId= - already initialized
2015-02-12 16:12:46,806 INFO  jvm.JvmMetrics (JvmMetrics.java:init(71)) - Cannot initialize JVM Metrics with processName=JobTracker, sessionId= - already initialized
2015-02-12 16:12:46,829 INFO  mapred.FileInputFormat (FileInputFormat.java:listStatus(247)) - Total input paths to process : 1
2015-02-12 16:12:46,846 INFO  mapreduce.JobSubmitter (JobSubmitter.java:submitJobInternal(492)) - number of splits:1
2015-02-12 16:12:46,860 INFO  mapreduce.JobSubmitter (JobSubmitter.java:printTokens(575)) - Submitting tokens for job: job_local2036438259_0004
2015-02-12 16:12:46,874 WARN  conf.Configuration (Configuration.java:loadProperty(2510)) - file:/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/tmp/cascading/staging/soberoi2036438259/.staging/job_local2036438259_0004/job.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.retry.interval;  Ignoring.
2015-02-12 16:12:46,875 WARN  conf.Configuration (Configuration.java:loadProperty(2510)) - file:/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/tmp/cascading/staging/soberoi2036438259/.staging/job_local2036438259_0004/job.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.attempts;  Ignoring.
2015-02-12 16:12:46,915 WARN  conf.Configuration (Configuration.java:loadProperty(2510)) - file:/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/test/tmp/mapred/local/localRunner/soberoi/job_local2036438259_0004/job_local2036438259_0004.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.retry.interval;  Ignoring.
2015-02-12 16:12:46,916 WARN  conf.Configuration (Configuration.java:loadProperty(2510)) - file:/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/test/tmp/mapred/local/localRunner/soberoi/job_local2036438259_0004/job_local2036438259_0004.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.attempts;  Ignoring.
2015-02-12 16:12:46,917 INFO  mapreduce.Job (Job.java:submit(1299)) - The url to track the job: http://localhost:8080/
2015-02-12 16:12:46,917 INFO  flow.FlowStep (BaseFlowStep.java:logInfo(834)) - [shape] submitted hadoop job: job_local2036438259_0004
2015-02-12 16:12:46,917 INFO  mapred.LocalJobRunner (LocalJobRunner.java:createOutputCommitter(471)) - OutputCommitter set in config null
2015-02-12 16:12:46,917 INFO  flow.FlowStep (BaseFlowStep.java:logInfo(834)) - [shape] tracking url: http://localhost:8080/
2015-02-12 16:12:46,917 INFO  mapred.LocalJobRunner (LocalJobRunner.java:createOutputCommitter(489)) - OutputCommitter is org.apache.hadoop.mapred.FileOutputCommitter
2015-02-12 16:12:46,919 INFO  mapred.LocalJobRunner (LocalJobRunner.java:runTasks(448)) - Waiting for map tasks
2015-02-12 16:12:46,919 INFO  mapred.LocalJobRunner (LocalJobRunner.java:run(224)) - Starting task: attempt_local2036438259_0004_m_000000_0
2015-02-12 16:12:46,920 INFO  util.ProcfsBasedProcessTree (ProcfsBasedProcessTree.java:isAvailable(181)) - ProcfsBasedProcessTree currently is supported only on Linux.
2015-02-12 16:12:46,921 INFO  mapred.Task (Task.java:initialize(587)) -  Using ResourceCalculatorProcessTree : null
2015-02-12 16:12:46,921 INFO  io.MultiInputSplit (MultiInputSplit.java:readFields(161)) - current split input path: file:/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/classes/test/data/lower.txt
2015-02-12 16:12:46,922 INFO  mapred.MapTask (MapTask.java:updateJobWithSplit(482)) - Processing split: cascading.tap.hadoop.io.MultiInputSplit@4a27dbfc
2015-02-12 16:12:46,922 INFO  mapred.MapTask (MapTask.java:runOldMapper(436)) - numReduceTasks: 0
2015-02-12 16:12:46,929 INFO  hadoop.FlowMapper (FlowMapper.java:configure(71)) - cascading version: 2.6.3
2015-02-12 16:12:46,929 INFO  hadoop.FlowMapper (FlowMapper.java:configure(72)) - child jvm opts: -Xmx200m
2015-02-12 16:12:46,933 INFO  hadoop.FlowMapper (FlowMapper.java:configure(87)) - sourcing from: Hfs["TextLine[['offset', 'line']->[ALL]]"]["/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/classes/test/data/lower.txt"]
2015-02-12 16:12:46,933 INFO  hadoop.FlowMapper (FlowMapper.java:configure(90)) - sinking to: Hfs["TextLine[['line']->['item', 'element']]"]["/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/test/output/hadoop2-mr1/assemblyhelpersplatform/renameall"]
2015-02-12 16:12:46,936 INFO  mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) - 
2015-02-12 16:12:46,936 INFO  mapred.Task (Task.java:done(1001)) - Task:attempt_local2036438259_0004_m_000000_0 is done. And is in the process of committing
2015-02-12 16:12:46,937 INFO  mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) - 
2015-02-12 16:12:46,937 INFO  mapred.Task (Task.java:commit(1162)) - Task attempt_local2036438259_0004_m_000000_0 is allowed to commit now
2015-02-12 16:12:46,938 INFO  output.FileOutputCommitter (FileOutputCommitter.java:commitTask(439)) - Saved output of task 'attempt_local2036438259_0004_m_000000_0' to file:/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/test/output/hadoop2-mr1/assemblyhelpersplatform/renameall/_temporary/0/task_local2036438259_0004_m_000000
2015-02-12 16:12:46,939 INFO  mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) - file:/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/classes/test/data/lower.txt:0+19
2015-02-12 16:12:46,939 INFO  mapred.Task (Task.java:sendDone(1121)) - Task 'attempt_local2036438259_0004_m_000000_0' done.
2015-02-12 16:12:46,939 INFO  mapred.LocalJobRunner (LocalJobRunner.java:run(249)) - Finishing task: attempt_local2036438259_0004_m_000000_0
2015-02-12 16:12:46,939 INFO  mapred.LocalJobRunner (LocalJobRunner.java:runTasks(456)) - map task executor complete.
2015-02-12 16:12:46,955 INFO  util.Hadoop18TapUtil (Hadoop18TapUtil.java:cleanTempPath(219)) - deleting temp path /Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/test/output/hadoop2-mr1/assemblyhelpersplatform/renameall/_temporary
2015-02-12 16:12:46,977 INFO  mapred.FileInputFormat (FileInputFormat.java:listStatus(247)) - Total input paths to process : 1
2015-02-12 16:12:46,978 INFO  cascading.PlatformTestCase (PlatformTestCase.java:tearDown(209)) - copying to local /Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/test/output//hadoop2-mr1/assemblyhelpersplatform/renameall
2015-02-12 16:12:46,993 INFO  util.HadoopUtil (HadoopUtil.java:findMainClass(336)) - using default application jar, may cause class not found exceptions on the cluster
2015-02-12 16:12:46,993 INFO  planner.HadoopPlanner (HadoopPlanner.java:initialize(225)) - using application jar: /Users/soberoi/.gradle/caches/modules-2/files-2.1/cascading/cascading-hadoop2-mr1/2.6.3/9ae016e2b8d246b9c1de9168e715d9f1b971aa4a/cascading-hadoop2-mr1-2.6.3.jar
2015-02-12 16:12:47,005 INFO  Configuration.deprecation (Configuration.java:warnOnceIfDeprecated(1022)) - mapred.used.genericoptionsparser is deprecated. Instead, use mapreduce.client.genericoptionsparser.used
2015-02-12 16:12:47,008 INFO  Configuration.deprecation (Configuration.java:warnOnceIfDeprecated(1022)) - mapred.job.tracker is deprecated. Instead, use mapreduce.jobtracker.address
2015-02-12 16:12:47,018 INFO  Configuration.deprecation (Configuration.java:warnOnceIfDeprecated(1022)) - mapred.output.compress is deprecated. Instead, use mapreduce.output.fileoutputformat.compress
2015-02-12 16:12:47,024 INFO  flow.Flow (BaseFlow.java:logInfo(1378)) - [count] starting
2015-02-12 16:12:47,024 INFO  flow.Flow (BaseFlow.java:logInfo(1378)) - [count]  source: Hfs["TextDelimited[['num', 'char']]"]["/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/classes/test/data/lhs.txt"]
2015-02-12 16:12:47,025 INFO  flow.Flow (BaseFlow.java:logInfo(1378)) - [count]  sink: Hfs["TextDelimited[['count', 'count2']]"]["/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/test/output/hadoop2-mr1/assemblyhelpersplatform/countcount"]
2015-02-12 16:12:47,025 INFO  flow.Flow (BaseFlow.java:logInfo(1378)) - [count]  parallel execution is enabled: false
2015-02-12 16:12:47,025 INFO  flow.Flow (BaseFlow.java:logInfo(1378)) - [count]  starting jobs: 2
2015-02-12 16:12:47,025 INFO  flow.Flow (BaseFlow.java:logInfo(1378)) - [count]  allocating threads: 1
2015-02-12 16:12:47,025 INFO  flow.FlowStep (BaseFlowStep.java:logInfo(834)) - [count] starting step: (1/2)
2015-02-12 16:12:47,026 INFO  jvm.JvmMetrics (JvmMetrics.java:init(71)) - Cannot initialize JVM Metrics with processName=JobTracker, sessionId= - already initialized
2015-02-12 16:12:47,027 INFO  jvm.JvmMetrics (JvmMetrics.java:init(71)) - Cannot initialize JVM Metrics with processName=JobTracker, sessionId= - already initialized
2015-02-12 16:12:47,051 INFO  mapred.FileInputFormat (FileInputFormat.java:listStatus(247)) - Total input paths to process : 1
2015-02-12 16:12:47,069 INFO  mapreduce.JobSubmitter (JobSubmitter.java:submitJobInternal(492)) - number of splits:1
2015-02-12 16:12:47,090 INFO  mapreduce.JobSubmitter (JobSubmitter.java:printTokens(575)) - Submitting tokens for job: job_local204371883_0005
2015-02-12 16:12:47,102 WARN  conf.Configuration (Configuration.java:loadProperty(2510)) - file:/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/tmp/cascading/staging/soberoi204371883/.staging/job_local204371883_0005/job.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.retry.interval;  Ignoring.
2015-02-12 16:12:47,103 WARN  conf.Configuration (Configuration.java:loadProperty(2510)) - file:/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/tmp/cascading/staging/soberoi204371883/.staging/job_local204371883_0005/job.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.attempts;  Ignoring.
2015-02-12 16:12:47,138 WARN  conf.Configuration (Configuration.java:loadProperty(2510)) - file:/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/test/tmp/mapred/local/localRunner/soberoi/job_local204371883_0005/job_local204371883_0005.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.retry.interval;  Ignoring.
2015-02-12 16:12:47,139 WARN  conf.Configuration (Configuration.java:loadProperty(2510)) - file:/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/test/tmp/mapred/local/localRunner/soberoi/job_local204371883_0005/job_local204371883_0005.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.attempts;  Ignoring.
2015-02-12 16:12:47,139 INFO  mapreduce.Job (Job.java:submit(1299)) - The url to track the job: http://localhost:8080/
2015-02-12 16:12:47,140 INFO  mapred.LocalJobRunner (LocalJobRunner.java:createOutputCommitter(471)) - OutputCommitter set in config null
2015-02-12 16:12:47,140 INFO  flow.FlowStep (BaseFlowStep.java:logInfo(834)) - [count] submitted hadoop job: job_local204371883_0005
2015-02-12 16:12:47,140 INFO  mapred.LocalJobRunner (LocalJobRunner.java:createOutputCommitter(489)) - OutputCommitter is org.apache.hadoop.mapred.FileOutputCommitter
2015-02-12 16:12:47,140 INFO  flow.FlowStep (BaseFlowStep.java:logInfo(834)) - [count] tracking url: http://localhost:8080/
2015-02-12 16:12:47,142 INFO  mapred.LocalJobRunner (LocalJobRunner.java:runTasks(448)) - Waiting for map tasks
2015-02-12 16:12:47,142 INFO  mapred.LocalJobRunner (LocalJobRunner.java:run(224)) - Starting task: attempt_local204371883_0005_m_000000_0
2015-02-12 16:12:47,144 INFO  util.ProcfsBasedProcessTree (ProcfsBasedProcessTree.java:isAvailable(181)) - ProcfsBasedProcessTree currently is supported only on Linux.
2015-02-12 16:12:47,144 INFO  mapred.Task (Task.java:initialize(587)) -  Using ResourceCalculatorProcessTree : null
2015-02-12 16:12:47,145 INFO  io.MultiInputSplit (MultiInputSplit.java:readFields(161)) - current split input path: file:/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/classes/test/data/lhs.txt
2015-02-12 16:12:47,145 INFO  mapred.MapTask (MapTask.java:updateJobWithSplit(482)) - Processing split: cascading.tap.hadoop.io.MultiInputSplit@7c4c3dc2
2015-02-12 16:12:47,146 INFO  mapred.MapTask (MapTask.java:runOldMapper(436)) - numReduceTasks: 1
2015-02-12 16:12:47,158 INFO  mapred.MapTask (MapTask.java:setEquator(1202)) - (EQUATOR) 0 kvi 26214396(104857584)
2015-02-12 16:12:47,158 INFO  mapred.MapTask (MapTask.java:init(995)) - mapreduce.task.io.sort.mb: 100
2015-02-12 16:12:47,158 INFO  mapred.MapTask (MapTask.java:init(996)) - soft limit at 83886080
2015-02-12 16:12:47,158 INFO  mapred.MapTask (MapTask.java:init(997)) - bufstart = 0; bufvoid = 104857600
2015-02-12 16:12:47,158 INFO  mapred.MapTask (MapTask.java:init(998)) - kvstart = 26214396; length = 6553600
2015-02-12 16:12:47,159 INFO  mapred.MapTask (MapTask.java:createSortingCollector(402)) - Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer
2015-02-12 16:12:47,159 INFO  hadoop.FlowMapper (FlowMapper.java:configure(71)) - cascading version: 2.6.3
2015-02-12 16:12:47,159 INFO  hadoop.FlowMapper (FlowMapper.java:configure(72)) - child jvm opts: -Xmx200m
2015-02-12 16:12:47,166 INFO  hadoop.FlowMapper (FlowMapper.java:configure(87)) - sourcing from: Hfs["TextDelimited[['num', 'char']]"]["/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/classes/test/data/lhs.txt"]
2015-02-12 16:12:47,166 INFO  hadoop.FlowMapper (FlowMapper.java:configure(90)) - sinking to: GroupBy(count)[by:[{1}:'char']]
2015-02-12 16:12:47,166 INFO  assembly.AggregateBy (AggregateBy.java:prepare(267)) - using threshold value: 2
2015-02-12 16:12:47,167 INFO  assembly.AggregateBy (AggregateBy.java:removeEldestEntry(327)) - flushed keys num times: 1, with threshold: 2
2015-02-12 16:12:47,167 INFO  assembly.AggregateBy (AggregateBy.java:removeEldestEntry(328)) - mem on flush (mb), free: 88, total: 509, max: 672
2015-02-12 16:12:47,167 INFO  assembly.AggregateBy (AggregateBy.java:removeEldestEntry(333)) - total mem is 75% of max mem, to better utilize unused memory consider increasing current LRU threshold with system property "cascading.aggregateby.threshold"
2015-02-12 16:12:47,168 INFO  assembly.AggregateBy (AggregateBy.java:removeEldestEntry(327)) - flushed keys num times: 3, with threshold: 2
2015-02-12 16:12:47,168 INFO  assembly.AggregateBy (AggregateBy.java:removeEldestEntry(328)) - mem on flush (mb), free: 88, total: 509, max: 672
2015-02-12 16:12:47,168 INFO  assembly.AggregateBy (AggregateBy.java:removeEldestEntry(333)) - total mem is 75% of max mem, to better utilize unused memory consider increasing current LRU threshold with system property "cascading.aggregateby.threshold"
2015-02-12 16:12:47,168 INFO  assembly.AggregateBy (AggregateBy.java:removeEldestEntry(327)) - flushed keys num times: 5, with threshold: 2
2015-02-12 16:12:47,168 INFO  assembly.AggregateBy (AggregateBy.java:removeEldestEntry(328)) - mem on flush (mb), free: 88, total: 509, max: 672
2015-02-12 16:12:47,169 INFO  assembly.AggregateBy (AggregateBy.java:removeEldestEntry(333)) - total mem is 75% of max mem, to better utilize unused memory consider increasing current LRU threshold with system property "cascading.aggregateby.threshold"
2015-02-12 16:12:47,169 INFO  assembly.AggregateBy (AggregateBy.java:removeEldestEntry(327)) - flushed keys num times: 7, with threshold: 2
2015-02-12 16:12:47,169 INFO  assembly.AggregateBy (AggregateBy.java:removeEldestEntry(328)) - mem on flush (mb), free: 88, total: 509, max: 672
2015-02-12 16:12:47,169 INFO  assembly.AggregateBy (AggregateBy.java:removeEldestEntry(333)) - total mem is 75% of max mem, to better utilize unused memory consider increasing current LRU threshold with system property "cascading.aggregateby.threshold"
2015-02-12 16:12:47,170 INFO  mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) - 
2015-02-12 16:12:47,170 INFO  mapred.MapTask (MapTask.java:flush(1457)) - Starting flush of map output
2015-02-12 16:12:47,170 INFO  mapred.MapTask (MapTask.java:flush(1475)) - Spilling map output
2015-02-12 16:12:47,170 INFO  mapred.MapTask (MapTask.java:flush(1476)) - bufstart = 0; bufend = 99; bufvoid = 104857600
2015-02-12 16:12:47,170 INFO  mapred.MapTask (MapTask.java:flush(1478)) - kvstart = 26214396(104857584); kvend = 26214364(104857456); length = 33/6553600
2015-02-12 16:12:47,172 INFO  mapred.MapTask (MapTask.java:sortAndSpill(1660)) - Finished spill 0
2015-02-12 16:12:47,174 INFO  mapred.Task (Task.java:done(1001)) - Task:attempt_local204371883_0005_m_000000_0 is done. And is in the process of committing
2015-02-12 16:12:47,175 INFO  mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) - file:/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/classes/test/data/lhs.txt:0+52
2015-02-12 16:12:47,175 INFO  mapred.Task (Task.java:sendDone(1121)) - Task 'attempt_local204371883_0005_m_000000_0' done.
2015-02-12 16:12:47,175 INFO  mapred.LocalJobRunner (LocalJobRunner.java:run(249)) - Finishing task: attempt_local204371883_0005_m_000000_0
2015-02-12 16:12:47,175 INFO  mapred.LocalJobRunner (LocalJobRunner.java:runTasks(456)) - map task executor complete.
2015-02-12 16:12:47,176 INFO  mapred.LocalJobRunner (LocalJobRunner.java:runTasks(448)) - Waiting for reduce tasks
2015-02-12 16:12:47,176 INFO  mapred.LocalJobRunner (LocalJobRunner.java:run(302)) - Starting task: attempt_local204371883_0005_r_000000_0
2015-02-12 16:12:47,177 INFO  util.ProcfsBasedProcessTree (ProcfsBasedProcessTree.java:isAvailable(181)) - ProcfsBasedProcessTree currently is supported only on Linux.
2015-02-12 16:12:47,177 INFO  mapred.Task (Task.java:initialize(587)) -  Using ResourceCalculatorProcessTree : null
2015-02-12 16:12:47,177 INFO  mapred.ReduceTask (ReduceTask.java:run(362)) - Using ShuffleConsumerPlugin: org.apache.hadoop.mapreduce.task.reduce.Shuffle@5d7566df
2015-02-12 16:12:47,178 INFO  reduce.MergeManagerImpl (MergeManagerImpl.java:<init>(195)) - MergerManager: memoryLimit=493250144, maxSingleShuffleLimit=123312536, mergeThreshold=325545120, ioSortFactor=10, memToMemMergeOutputsThreshold=10
2015-02-12 16:12:47,179 INFO  reduce.EventFetcher (EventFetcher.java:run(61)) - attempt_local204371883_0005_r_000000_0 Thread started: EventFetcher for fetching Map Completion Events
2015-02-12 16:12:47,180 INFO  reduce.LocalFetcher (LocalFetcher.java:copyMapOutput(141)) - localfetcher#4 about to shuffle output of map attempt_local204371883_0005_m_000000_0 decomp: 119 len: 123 to MEMORY
2015-02-12 16:12:47,180 INFO  reduce.InMemoryMapOutput (InMemoryMapOutput.java:shuffle(100)) - Read 119 bytes from map-output for attempt_local204371883_0005_m_000000_0
2015-02-12 16:12:47,180 INFO  reduce.MergeManagerImpl (MergeManagerImpl.java:closeInMemoryFile(313)) - closeInMemoryFile -> map-output of size: 119, inMemoryMapOutputs.size() -> 1, commitMemory -> 0, usedMemory ->119
2015-02-12 16:12:47,181 INFO  reduce.EventFetcher (EventFetcher.java:run(76)) - EventFetcher is interrupted.. Returning
2015-02-12 16:12:47,181 INFO  mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) - 1 / 1 copied.
2015-02-12 16:12:47,182 INFO  reduce.MergeManagerImpl (MergeManagerImpl.java:finalMerge(673)) - finalMerge called with 1 in-memory map-outputs and 0 on-disk map-outputs
2015-02-12 16:12:47,183 INFO  mapred.Merger (Merger.java:merge(606)) - Merging 1 sorted segments
2015-02-12 16:12:47,184 INFO  mapred.Merger (Merger.java:merge(705)) - Down to the last merge-pass, with 1 segments left of total size: 110 bytes
2015-02-12 16:12:47,185 INFO  reduce.MergeManagerImpl (MergeManagerImpl.java:finalMerge(750)) - Merged 1 segments, 119 bytes to disk to satisfy reduce memory limit
2015-02-12 16:12:47,185 INFO  reduce.MergeManagerImpl (MergeManagerImpl.java:finalMerge(780)) - Merging 1 files, 123 bytes from disk
2015-02-12 16:12:47,185 INFO  reduce.MergeManagerImpl (MergeManagerImpl.java:finalMerge(795)) - Merging 0 segments, 0 bytes from memory into reduce
2015-02-12 16:12:47,185 INFO  mapred.Merger (Merger.java:merge(606)) - Merging 1 sorted segments
2015-02-12 16:12:47,186 INFO  mapred.Merger (Merger.java:merge(705)) - Down to the last merge-pass, with 1 segments left of total size: 110 bytes
2015-02-12 16:12:47,186 INFO  mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) - 1 / 1 copied.
2015-02-12 16:12:47,186 INFO  hadoop.FlowReducer (FlowReducer.java:configure(78)) - cascading version: 2.6.3
2015-02-12 16:12:47,187 INFO  hadoop.FlowReducer (FlowReducer.java:configure(79)) - child jvm opts: -Xmx200m
2015-02-12 16:12:47,193 INFO  hadoop.FlowReducer (FlowReducer.java:configure(97)) - sourcing from: GroupBy(count)[by:[{1}:'char']]
2015-02-12 16:12:47,193 INFO  hadoop.FlowReducer (FlowReducer.java:configure(100)) - sinking to: TempHfs["SequenceFile[['count', 'count2']]"][7450648248/count/]
2015-02-12 16:12:47,207 INFO  assembly.AggregateBy (AggregateBy.java:prepare(267)) - using threshold value: 2
2015-02-12 16:12:47,209 INFO  assembly.AggregateBy (AggregateBy.java:removeEldestEntry(327)) - flushed keys num times: 1, with threshold: 2
2015-02-12 16:12:47,209 INFO  assembly.AggregateBy (AggregateBy.java:removeEldestEntry(328)) - mem on flush (mb), free: 77, total: 509, max: 672
2015-02-12 16:12:47,209 INFO  assembly.AggregateBy (AggregateBy.java:removeEldestEntry(333)) - total mem is 75% of max mem, to better utilize unused memory consider increasing current LRU threshold with system property "cascading.aggregateby.threshold"
2015-02-12 16:12:47,210 INFO  mapred.Task (Task.java:done(1001)) - Task:attempt_local204371883_0005_r_000000_0 is done. And is in the process of committing
2015-02-12 16:12:47,211 INFO  mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) - 1 / 1 copied.
2015-02-12 16:12:47,211 INFO  mapred.Task (Task.java:commit(1162)) - Task attempt_local204371883_0005_r_000000_0 is allowed to commit now
2015-02-12 16:12:47,212 INFO  output.FileOutputCommitter (FileOutputCommitter.java:commitTask(439)) - Saved output of task 'attempt_local204371883_0005_r_000000_0' to file:/tmp/hadoop-soberoi/7450648248_count_7A613910A9AE40EF949D3719A07AE1BF/_temporary/0/task_local204371883_0005_r_000000
2015-02-12 16:12:47,213 INFO  mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) - reduce > reduce
2015-02-12 16:12:47,213 INFO  mapred.Task (Task.java:sendDone(1121)) - Task 'attempt_local204371883_0005_r_000000_0' done.
2015-02-12 16:12:47,213 INFO  mapred.LocalJobRunner (LocalJobRunner.java:run(325)) - Finishing task: attempt_local204371883_0005_r_000000_0
2015-02-12 16:12:47,213 INFO  mapred.LocalJobRunner (LocalJobRunner.java:runTasks(456)) - reduce task executor complete.
2015-02-12 16:12:47,233 INFO  flow.FlowStep (BaseFlowStep.java:logInfo(834)) - [count] starting step: (2/2) ...elpersplatform/countcount
2015-02-12 16:12:47,234 INFO  jvm.JvmMetrics (JvmMetrics.java:init(71)) - Cannot initialize JVM Metrics with processName=JobTracker, sessionId= - already initialized
2015-02-12 16:12:47,235 INFO  jvm.JvmMetrics (JvmMetrics.java:init(71)) - Cannot initialize JVM Metrics with processName=JobTracker, sessionId= - already initialized
2015-02-12 16:12:47,282 INFO  mapred.FileInputFormat (FileInputFormat.java:listStatus(247)) - Total input paths to process : 1
2015-02-12 16:12:47,301 INFO  mapreduce.JobSubmitter (JobSubmitter.java:submitJobInternal(492)) - number of splits:1
2015-02-12 16:12:47,311 INFO  Configuration.deprecation (Configuration.java:warnOnceIfDeprecated(1022)) - mapred.used.genericoptionsparser is deprecated. Instead, use mapreduce.client.genericoptionsparser.used
2015-02-12 16:12:47,311 INFO  Configuration.deprecation (Configuration.java:warnOnceIfDeprecated(1022)) - mapred.job.tracker is deprecated. Instead, use mapreduce.jobtracker.address
2015-02-12 16:12:47,315 INFO  mapreduce.JobSubmitter (JobSubmitter.java:printTokens(575)) - Submitting tokens for job: job_local1176816738_0006
2015-02-12 16:12:47,329 WARN  conf.Configuration (Configuration.java:loadProperty(2510)) - file:/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/tmp/cascading/staging/soberoi1176816738/.staging/job_local1176816738_0006/job.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.retry.interval;  Ignoring.
2015-02-12 16:12:47,330 WARN  conf.Configuration (Configuration.java:loadProperty(2510)) - file:/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/tmp/cascading/staging/soberoi1176816738/.staging/job_local1176816738_0006/job.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.attempts;  Ignoring.
2015-02-12 16:12:47,362 WARN  conf.Configuration (Configuration.java:loadProperty(2510)) - file:/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/test/tmp/mapred/local/localRunner/soberoi/job_local1176816738_0006/job_local1176816738_0006.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.retry.interval;  Ignoring.
2015-02-12 16:12:47,363 WARN  conf.Configuration (Configuration.java:loadProperty(2510)) - file:/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/test/tmp/mapred/local/localRunner/soberoi/job_local1176816738_0006/job_local1176816738_0006.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.attempts;  Ignoring.
2015-02-12 16:12:47,364 INFO  mapreduce.Job (Job.java:submit(1299)) - The url to track the job: http://localhost:8080/
2015-02-12 16:12:47,364 INFO  mapred.LocalJobRunner (LocalJobRunner.java:createOutputCommitter(471)) - OutputCommitter set in config null
2015-02-12 16:12:47,364 INFO  mapred.LocalJobRunner (LocalJobRunner.java:createOutputCommitter(489)) - OutputCommitter is org.apache.hadoop.mapred.FileOutputCommitter
2015-02-12 16:12:47,364 INFO  flow.FlowStep (BaseFlowStep.java:logInfo(834)) - [count] submitted hadoop job: job_local1176816738_0006
2015-02-12 16:12:47,365 INFO  flow.FlowStep (BaseFlowStep.java:logInfo(834)) - [count] tracking url: http://localhost:8080/
2015-02-12 16:12:47,367 INFO  mapred.LocalJobRunner (LocalJobRunner.java:runTasks(448)) - Waiting for map tasks
2015-02-12 16:12:47,367 INFO  mapred.LocalJobRunner (LocalJobRunner.java:run(224)) - Starting task: attempt_local1176816738_0006_m_000000_0
2015-02-12 16:12:47,368 INFO  util.ProcfsBasedProcessTree (ProcfsBasedProcessTree.java:isAvailable(181)) - ProcfsBasedProcessTree currently is supported only on Linux.
2015-02-12 16:12:47,368 INFO  mapred.Task (Task.java:initialize(587)) -  Using ResourceCalculatorProcessTree : null
2015-02-12 16:12:47,370 INFO  io.MultiInputSplit (MultiInputSplit.java:readFields(161)) - current split input path: file:/tmp/hadoop-soberoi/7450648248_count_7A613910A9AE40EF949D3719A07AE1BF/part-00000
2015-02-12 16:12:47,370 INFO  mapred.MapTask (MapTask.java:updateJobWithSplit(482)) - Processing split: cascading.tap.hadoop.io.MultiInputSplit@346fadf4
2015-02-12 16:12:47,376 INFO  mapred.MapTask (MapTask.java:runOldMapper(436)) - numReduceTasks: 1
2015-02-12 16:12:47,390 INFO  mapred.MapTask (MapTask.java:setEquator(1202)) - (EQUATOR) 0 kvi 26214396(104857584)
2015-02-12 16:12:47,390 INFO  mapred.MapTask (MapTask.java:init(995)) - mapreduce.task.io.sort.mb: 100
2015-02-12 16:12:47,390 INFO  mapred.MapTask (MapTask.java:init(996)) - soft limit at 83886080
2015-02-12 16:12:47,390 INFO  mapred.MapTask (MapTask.java:init(997)) - bufstart = 0; bufvoid = 104857600
2015-02-12 16:12:47,390 INFO  mapred.MapTask (MapTask.java:init(998)) - kvstart = 26214396; length = 6553600
2015-02-12 16:12:47,391 INFO  mapred.MapTask (MapTask.java:createSortingCollector(402)) - Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer
2015-02-12 16:12:47,391 INFO  hadoop.FlowMapper (FlowMapper.java:configure(71)) - cascading version: 2.6.3
2015-02-12 16:12:47,391 INFO  hadoop.FlowMapper (FlowMapper.java:configure(72)) - child jvm opts: -Xmx200m
2015-02-12 16:12:47,396 INFO  hadoop.FlowMapper (FlowMapper.java:configure(87)) - sourcing from: TempHfs["SequenceFile[['count', 'count2']]"][7450648248/count/]
2015-02-12 16:12:47,397 INFO  hadoop.FlowMapper (FlowMapper.java:configure(90)) - sinking to: GroupBy(count)[by:[{1}:'count']]
2015-02-12 16:12:47,398 INFO  mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) - 
2015-02-12 16:12:47,398 INFO  mapred.MapTask (MapTask.java:flush(1457)) - Starting flush of map output
2015-02-12 16:12:47,398 INFO  mapred.MapTask (MapTask.java:flush(1475)) - Spilling map output
2015-02-12 16:12:47,398 INFO  mapred.MapTask (MapTask.java:flush(1476)) - bufstart = 0; bufend = 21; bufvoid = 104857600
2015-02-12 16:12:47,398 INFO  mapred.MapTask (MapTask.java:flush(1478)) - kvstart = 26214396(104857584); kvend = 26214388(104857552); length = 9/6553600
2015-02-12 16:12:47,400 INFO  mapred.MapTask (MapTask.java:sortAndSpill(1660)) - Finished spill 0
2015-02-12 16:12:47,402 INFO  mapred.Task (Task.java:done(1001)) - Task:attempt_local1176816738_0006_m_000000_0 is done. And is in the process of committing
2015-02-12 16:12:47,403 INFO  mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) - file:/tmp/hadoop-soberoi/7450648248_count_7A613910A9AE40EF949D3719A07AE1BF/part-00000:0+112
2015-02-12 16:12:47,403 INFO  mapred.Task (Task.java:sendDone(1121)) - Task 'attempt_local1176816738_0006_m_000000_0' done.
2015-02-12 16:12:47,404 INFO  mapred.LocalJobRunner (LocalJobRunner.java:run(249)) - Finishing task: attempt_local1176816738_0006_m_000000_0
2015-02-12 16:12:47,404 INFO  mapred.LocalJobRunner (LocalJobRunner.java:runTasks(456)) - map task executor complete.
2015-02-12 16:12:47,404 INFO  mapred.LocalJobRunner (LocalJobRunner.java:runTasks(448)) - Waiting for reduce tasks
2015-02-12 16:12:47,404 INFO  mapred.LocalJobRunner (LocalJobRunner.java:run(302)) - Starting task: attempt_local1176816738_0006_r_000000_0
2015-02-12 16:12:47,406 INFO  util.ProcfsBasedProcessTree (ProcfsBasedProcessTree.java:isAvailable(181)) - ProcfsBasedProcessTree currently is supported only on Linux.
2015-02-12 16:12:47,406 INFO  mapred.Task (Task.java:initialize(587)) -  Using ResourceCalculatorProcessTree : null
2015-02-12 16:12:47,406 INFO  mapred.ReduceTask (ReduceTask.java:run(362)) - Using ShuffleConsumerPlugin: org.apache.hadoop.mapreduce.task.reduce.Shuffle@4fe087e7
2015-02-12 16:12:47,407 INFO  reduce.MergeManagerImpl (MergeManagerImpl.java:<init>(195)) - MergerManager: memoryLimit=493250144, maxSingleShuffleLimit=123312536, mergeThreshold=325545120, ioSortFactor=10, memToMemMergeOutputsThreshold=10
2015-02-12 16:12:47,407 INFO  reduce.EventFetcher (EventFetcher.java:run(61)) - attempt_local1176816738_0006_r_000000_0 Thread started: EventFetcher for fetching Map Completion Events
2015-02-12 16:12:47,409 INFO  reduce.LocalFetcher (LocalFetcher.java:copyMapOutput(141)) - localfetcher#5 about to shuffle output of map attempt_local1176816738_0006_m_000000_0 decomp: 29 len: 33 to MEMORY
2015-02-12 16:12:47,409 INFO  reduce.InMemoryMapOutput (InMemoryMapOutput.java:shuffle(100)) - Read 29 bytes from map-output for attempt_local1176816738_0006_m_000000_0
2015-02-12 16:12:47,409 INFO  reduce.MergeManagerImpl (MergeManagerImpl.java:closeInMemoryFile(313)) - closeInMemoryFile -> map-output of size: 29, inMemoryMapOutputs.size() -> 1, commitMemory -> 0, usedMemory ->29
2015-02-12 16:12:47,410 INFO  reduce.EventFetcher (EventFetcher.java:run(76)) - EventFetcher is interrupted.. Returning
2015-02-12 16:12:47,411 INFO  mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) - 1 / 1 copied.
2015-02-12 16:12:47,411 INFO  reduce.MergeManagerImpl (MergeManagerImpl.java:finalMerge(673)) - finalMerge called with 1 in-memory map-outputs and 0 on-disk map-outputs
2015-02-12 16:12:47,413 INFO  mapred.Merger (Merger.java:merge(606)) - Merging 1 sorted segments
2015-02-12 16:12:47,413 INFO  mapred.Merger (Merger.java:merge(705)) - Down to the last merge-pass, with 1 segments left of total size: 24 bytes
2015-02-12 16:12:47,415 INFO  reduce.MergeManagerImpl (MergeManagerImpl.java:finalMerge(750)) - Merged 1 segments, 29 bytes to disk to satisfy reduce memory limit
2015-02-12 16:12:47,415 INFO  reduce.MergeManagerImpl (MergeManagerImpl.java:finalMerge(780)) - Merging 1 files, 33 bytes from disk
2015-02-12 16:12:47,415 INFO  reduce.MergeManagerImpl (MergeManagerImpl.java:finalMerge(795)) - Merging 0 segments, 0 bytes from memory into reduce
2015-02-12 16:12:47,415 INFO  mapred.Merger (Merger.java:merge(606)) - Merging 1 sorted segments
2015-02-12 16:12:47,416 INFO  mapred.Merger (Merger.java:merge(705)) - Down to the last merge-pass, with 1 segments left of total size: 24 bytes
2015-02-12 16:12:47,416 INFO  mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) - 1 / 1 copied.
2015-02-12 16:12:47,416 INFO  hadoop.FlowReducer (FlowReducer.java:configure(78)) - cascading version: 2.6.3
2015-02-12 16:12:47,417 INFO  hadoop.FlowReducer (FlowReducer.java:configure(79)) - child jvm opts: -Xmx200m
2015-02-12 16:12:47,422 INFO  hadoop.FlowReducer (FlowReducer.java:configure(97)) - sourcing from: GroupBy(count)[by:[{1}:'count']]
2015-02-12 16:12:47,422 INFO  hadoop.FlowReducer (FlowReducer.java:configure(100)) - sinking to: Hfs["TextDelimited[['count', 'count2']]"]["/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/test/output/hadoop2-mr1/assemblyhelpersplatform/countcount"]
2015-02-12 16:12:47,431 INFO  mapred.Task (Task.java:done(1001)) - Task:attempt_local1176816738_0006_r_000000_0 is done. And is in the process of committing
2015-02-12 16:12:47,432 INFO  mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) - 1 / 1 copied.
2015-02-12 16:12:47,432 INFO  mapred.Task (Task.java:commit(1162)) - Task attempt_local1176816738_0006_r_000000_0 is allowed to commit now
2015-02-12 16:12:47,433 INFO  output.FileOutputCommitter (FileOutputCommitter.java:commitTask(439)) - Saved output of task 'attempt_local1176816738_0006_r_000000_0' to file:/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/test/output/hadoop2-mr1/assemblyhelpersplatform/countcount/_temporary/0/task_local1176816738_0006_r_000000
2015-02-12 16:12:47,434 INFO  mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) - reduce > reduce
2015-02-12 16:12:47,434 INFO  mapred.Task (Task.java:sendDone(1121)) - Task 'attempt_local1176816738_0006_r_000000_0' done.
2015-02-12 16:12:47,434 INFO  mapred.LocalJobRunner (LocalJobRunner.java:run(325)) - Finishing task: attempt_local1176816738_0006_r_000000_0
2015-02-12 16:12:47,434 INFO  mapred.LocalJobRunner (LocalJobRunner.java:runTasks(456)) - reduce task executor complete.
2015-02-12 16:12:47,454 INFO  util.Hadoop18TapUtil (Hadoop18TapUtil.java:cleanTempPath(219)) - deleting temp path /Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/test/output/hadoop2-mr1/assemblyhelpersplatform/countcount/_temporary
2015-02-12 16:12:47,476 INFO  mapred.FileInputFormat (FileInputFormat.java:listStatus(247)) - Total input paths to process : 1
2015-02-12 16:12:47,498 INFO  mapred.FileInputFormat (FileInputFormat.java:listStatus(247)) - Total input paths to process : 1
2015-02-12 16:12:47,499 INFO  cascading.PlatformTestCase (PlatformTestCase.java:tearDown(209)) - copying to local /Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/test/output//hadoop2-mr1/assemblyhelpersplatform/countcount
2015-02-12 16:12:47,515 INFO  util.HadoopUtil (HadoopUtil.java:findMainClass(336)) - using default application jar, may cause class not found exceptions on the cluster
2015-02-12 16:12:47,516 INFO  planner.HadoopPlanner (HadoopPlanner.java:initialize(225)) - using application jar: /Users/soberoi/.gradle/caches/modules-2/files-2.1/cascading/cascading-hadoop2-mr1/2.6.3/9ae016e2b8d246b9c1de9168e715d9f1b971aa4a/cascading-hadoop2-mr1-2.6.3.jar
2015-02-12 16:12:47,526 INFO  Configuration.deprecation (Configuration.java:warnOnceIfDeprecated(1022)) - mapred.used.genericoptionsparser is deprecated. Instead, use mapreduce.client.genericoptionsparser.used
2015-02-12 16:12:47,529 INFO  Configuration.deprecation (Configuration.java:warnOnceIfDeprecated(1022)) - mapred.job.tracker is deprecated. Instead, use mapreduce.jobtracker.address
2015-02-12 16:12:47,530 INFO  Configuration.deprecation (Configuration.java:warnOnceIfDeprecated(1022)) - mapred.output.compress is deprecated. Instead, use mapreduce.output.fileoutputformat.compress
2015-02-12 16:12:47,537 INFO  flow.Flow (BaseFlow.java:logInfo(1378)) - [count-lhs+count-rhs] starting
2015-02-12 16:12:47,538 INFO  flow.Flow (BaseFlow.java:logInfo(1378)) - [count-lhs+count-rhs]  source: Hfs["TextDelimited[['num', 'char']]"]["/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/classes/test/data/rhs.txt"]
2015-02-12 16:12:47,538 INFO  flow.Flow (BaseFlow.java:logInfo(1378)) - [count-lhs+count-rhs]  source: Hfs["TextDelimited[['num', 'char']]"]["/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/classes/test/data/lhs.txt"]
2015-02-12 16:12:47,538 INFO  flow.Flow (BaseFlow.java:logInfo(1378)) - [count-lhs+count-rhs]  sink: Hfs["TextDelimited[['char', 'count']]"]["/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/test/output/hadoop2-mr1/assemblyhelpersplatform/mergecount"]
2015-02-12 16:12:47,538 INFO  flow.Flow (BaseFlow.java:logInfo(1378)) - [count-lhs+count-rhs]  parallel execution is enabled: false
2015-02-12 16:12:47,538 INFO  flow.Flow (BaseFlow.java:logInfo(1378)) - [count-lhs+count-rhs]  starting jobs: 1
2015-02-12 16:12:47,539 INFO  flow.Flow (BaseFlow.java:logInfo(1378)) - [count-lhs+count-rhs]  allocating threads: 1
2015-02-12 16:12:47,539 INFO  flow.FlowStep (BaseFlowStep.java:logInfo(834)) - [count-lhs+count-rhs] starting step: (1/1) ...elpersplatform/mergecount
2015-02-12 16:12:47,540 INFO  jvm.JvmMetrics (JvmMetrics.java:init(71)) - Cannot initialize JVM Metrics with processName=JobTracker, sessionId= - already initialized
2015-02-12 16:12:47,540 INFO  jvm.JvmMetrics (JvmMetrics.java:init(71)) - Cannot initialize JVM Metrics with processName=JobTracker, sessionId= - already initialized
2015-02-12 16:12:47,564 INFO  mapred.FileInputFormat (FileInputFormat.java:listStatus(247)) - Total input paths to process : 1
2015-02-12 16:12:47,565 INFO  mapred.FileInputFormat (FileInputFormat.java:listStatus(247)) - Total input paths to process : 1
2015-02-12 16:12:47,582 INFO  mapreduce.JobSubmitter (JobSubmitter.java:submitJobInternal(492)) - number of splits:2
2015-02-12 16:12:47,595 INFO  mapreduce.JobSubmitter (JobSubmitter.java:printTokens(575)) - Submitting tokens for job: job_local154647584_0007
2015-02-12 16:12:47,607 WARN  conf.Configuration (Configuration.java:loadProperty(2510)) - file:/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/tmp/cascading/staging/soberoi154647584/.staging/job_local154647584_0007/job.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.retry.interval;  Ignoring.
2015-02-12 16:12:47,607 WARN  conf.Configuration (Configuration.java:loadProperty(2510)) - file:/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/tmp/cascading/staging/soberoi154647584/.staging/job_local154647584_0007/job.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.attempts;  Ignoring.
2015-02-12 16:12:47,641 WARN  conf.Configuration (Configuration.java:loadProperty(2510)) - file:/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/test/tmp/mapred/local/localRunner/soberoi/job_local154647584_0007/job_local154647584_0007.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.retry.interval;  Ignoring.
2015-02-12 16:12:47,642 WARN  conf.Configuration (Configuration.java:loadProperty(2510)) - file:/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/test/tmp/mapred/local/localRunner/soberoi/job_local154647584_0007/job_local154647584_0007.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.attempts;  Ignoring.
2015-02-12 16:12:47,642 INFO  mapreduce.Job (Job.java:submit(1299)) - The url to track the job: http://localhost:8080/
2015-02-12 16:12:47,642 INFO  flow.FlowStep (BaseFlowStep.java:logInfo(834)) - [count-lhs+count-rhs] submitted hadoop job: job_local154647584_0007
2015-02-12 16:12:47,642 INFO  mapred.LocalJobRunner (LocalJobRunner.java:createOutputCommitter(471)) - OutputCommitter set in config null
2015-02-12 16:12:47,642 INFO  flow.FlowStep (BaseFlowStep.java:logInfo(834)) - [count-lhs+count-rhs] tracking url: http://localhost:8080/
2015-02-12 16:12:47,643 INFO  mapred.LocalJobRunner (LocalJobRunner.java:createOutputCommitter(489)) - OutputCommitter is org.apache.hadoop.mapred.FileOutputCommitter
2015-02-12 16:12:47,645 INFO  mapred.LocalJobRunner (LocalJobRunner.java:runTasks(448)) - Waiting for map tasks
2015-02-12 16:12:47,645 INFO  mapred.LocalJobRunner (LocalJobRunner.java:run(224)) - Starting task: attempt_local154647584_0007_m_000000_0
2015-02-12 16:12:47,646 INFO  util.ProcfsBasedProcessTree (ProcfsBasedProcessTree.java:isAvailable(181)) - ProcfsBasedProcessTree currently is supported only on Linux.
2015-02-12 16:12:47,646 INFO  mapred.Task (Task.java:initialize(587)) -  Using ResourceCalculatorProcessTree : null
2015-02-12 16:12:47,647 INFO  io.MultiInputSplit (MultiInputSplit.java:readFields(161)) - current split input path: file:/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/classes/test/data/lhs.txt
2015-02-12 16:12:47,647 INFO  mapred.MapTask (MapTask.java:updateJobWithSplit(482)) - Processing split: cascading.tap.hadoop.io.MultiInputSplit@27ea4128
2015-02-12 16:12:47,649 INFO  mapred.MapTask (MapTask.java:runOldMapper(436)) - numReduceTasks: 1
2015-02-12 16:12:47,671 INFO  mapred.MapTask (MapTask.java:setEquator(1202)) - (EQUATOR) 0 kvi 26214396(104857584)
2015-02-12 16:12:47,672 INFO  mapred.MapTask (MapTask.java:init(995)) - mapreduce.task.io.sort.mb: 100
2015-02-12 16:12:47,673 INFO  mapred.MapTask (MapTask.java:init(996)) - soft limit at 83886080
2015-02-12 16:12:47,673 INFO  mapred.MapTask (MapTask.java:init(997)) - bufstart = 0; bufvoid = 104857600
2015-02-12 16:12:47,673 INFO  mapred.MapTask (MapTask.java:init(998)) - kvstart = 26214396; length = 6553600
2015-02-12 16:12:47,674 INFO  mapred.MapTask (MapTask.java:createSortingCollector(402)) - Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer
2015-02-12 16:12:47,674 INFO  hadoop.FlowMapper (FlowMapper.java:configure(71)) - cascading version: 2.6.3
2015-02-12 16:12:47,674 INFO  hadoop.FlowMapper (FlowMapper.java:configure(72)) - child jvm opts: -Xmx200m
2015-02-12 16:12:47,681 INFO  hadoop.FlowMapper (FlowMapper.java:configure(87)) - sourcing from: Hfs["TextDelimited[['num', 'char']]"]["/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/classes/test/data/lhs.txt"]
2015-02-12 16:12:47,681 INFO  hadoop.FlowMapper (FlowMapper.java:configure(90)) - sinking to: GroupBy(count-lhs+count-rhs)[by: count-lhs:[{1}:'char'] count-rhs:[{1}:'char']]
2015-02-12 16:12:47,681 INFO  assembly.AggregateBy (AggregateBy.java:prepare(267)) - using threshold value: 2
2015-02-12 16:12:47,682 INFO  assembly.AggregateBy (AggregateBy.java:removeEldestEntry(327)) - flushed keys num times: 1, with threshold: 2
2015-02-12 16:12:47,682 INFO  assembly.AggregateBy (AggregateBy.java:removeEldestEntry(328)) - mem on flush (mb), free: 192, total: 502, max: 672
2015-02-12 16:12:47,682 INFO  assembly.AggregateBy (AggregateBy.java:removeEldestEntry(333)) - total mem is 74% of max mem, to better utilize unused memory consider increasing current LRU threshold with system property "cascading.aggregateby.threshold"
2015-02-12 16:12:47,683 INFO  assembly.AggregateBy (AggregateBy.java:removeEldestEntry(327)) - flushed keys num times: 3, with threshold: 2
2015-02-12 16:12:47,683 INFO  assembly.AggregateBy (AggregateBy.java:removeEldestEntry(328)) - mem on flush (mb), free: 192, total: 502, max: 672
2015-02-12 16:12:47,683 INFO  assembly.AggregateBy (AggregateBy.java:removeEldestEntry(333)) - total mem is 74% of max mem, to better utilize unused memory consider increasing current LRU threshold with system property "cascading.aggregateby.threshold"
2015-02-12 16:12:47,683 INFO  assembly.AggregateBy (AggregateBy.java:removeEldestEntry(327)) - flushed keys num times: 5, with threshold: 2
2015-02-12 16:12:47,683 INFO  assembly.AggregateBy (AggregateBy.java:removeEldestEntry(328)) - mem on flush (mb), free: 192, total: 502, max: 672
2015-02-12 16:12:47,683 INFO  assembly.AggregateBy (AggregateBy.java:removeEldestEntry(333)) - total mem is 74% of max mem, to better utilize unused memory consider increasing current LRU threshold with system property "cascading.aggregateby.threshold"
2015-02-12 16:12:47,683 INFO  assembly.AggregateBy (AggregateBy.java:removeEldestEntry(327)) - flushed keys num times: 7, with threshold: 2
2015-02-12 16:12:47,684 INFO  assembly.AggregateBy (AggregateBy.java:removeEldestEntry(328)) - mem on flush (mb), free: 192, total: 502, max: 672
2015-02-12 16:12:47,684 INFO  assembly.AggregateBy (AggregateBy.java:removeEldestEntry(333)) - total mem is 74% of max mem, to better utilize unused memory consider increasing current LRU threshold with system property "cascading.aggregateby.threshold"
2015-02-12 16:12:47,685 INFO  mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) - 
2015-02-12 16:12:47,685 INFO  mapred.MapTask (MapTask.java:flush(1457)) - Starting flush of map output
2015-02-12 16:12:47,685 INFO  mapred.MapTask (MapTask.java:flush(1475)) - Spilling map output
2015-02-12 16:12:47,685 INFO  mapred.MapTask (MapTask.java:flush(1476)) - bufstart = 0; bufend = 99; bufvoid = 104857600
2015-02-12 16:12:47,685 INFO  mapred.MapTask (MapTask.java:flush(1478)) - kvstart = 26214396(104857584); kvend = 26214364(104857456); length = 33/6553600
2015-02-12 16:12:47,687 INFO  mapred.MapTask (MapTask.java:sortAndSpill(1660)) - Finished spill 0
2015-02-12 16:12:47,689 INFO  mapred.Task (Task.java:done(1001)) - Task:attempt_local154647584_0007_m_000000_0 is done. And is in the process of committing
2015-02-12 16:12:47,690 INFO  mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) - file:/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/classes/test/data/lhs.txt:0+52
2015-02-12 16:12:47,690 INFO  mapred.Task (Task.java:sendDone(1121)) - Task 'attempt_local154647584_0007_m_000000_0' done.
2015-02-12 16:12:47,690 INFO  mapred.LocalJobRunner (LocalJobRunner.java:run(249)) - Finishing task: attempt_local154647584_0007_m_000000_0
2015-02-12 16:12:47,690 INFO  mapred.LocalJobRunner (LocalJobRunner.java:run(224)) - Starting task: attempt_local154647584_0007_m_000001_0
2015-02-12 16:12:47,691 INFO  util.ProcfsBasedProcessTree (ProcfsBasedProcessTree.java:isAvailable(181)) - ProcfsBasedProcessTree currently is supported only on Linux.
2015-02-12 16:12:47,691 INFO  mapred.Task (Task.java:initialize(587)) -  Using ResourceCalculatorProcessTree : null
2015-02-12 16:12:47,692 INFO  io.MultiInputSplit (MultiInputSplit.java:readFields(161)) - current split input path: file:/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/classes/test/data/rhs.txt
2015-02-12 16:12:47,692 INFO  mapred.MapTask (MapTask.java:updateJobWithSplit(482)) - Processing split: cascading.tap.hadoop.io.MultiInputSplit@1d5ec199
2015-02-12 16:12:47,693 INFO  mapred.MapTask (MapTask.java:runOldMapper(436)) - numReduceTasks: 1
2015-02-12 16:12:47,705 INFO  mapred.MapTask (MapTask.java:setEquator(1202)) - (EQUATOR) 0 kvi 26214396(104857584)
2015-02-12 16:12:47,705 INFO  mapred.MapTask (MapTask.java:init(995)) - mapreduce.task.io.sort.mb: 100
2015-02-12 16:12:47,705 INFO  mapred.MapTask (MapTask.java:init(996)) - soft limit at 83886080
2015-02-12 16:12:47,705 INFO  mapred.MapTask (MapTask.java:init(997)) - bufstart = 0; bufvoid = 104857600
2015-02-12 16:12:47,705 INFO  mapred.MapTask (MapTask.java:init(998)) - kvstart = 26214396; length = 6553600
2015-02-12 16:12:47,706 INFO  mapred.MapTask (MapTask.java:createSortingCollector(402)) - Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer
2015-02-12 16:12:47,706 INFO  hadoop.FlowMapper (FlowMapper.java:configure(71)) - cascading version: 2.6.3
2015-02-12 16:12:47,706 INFO  hadoop.FlowMapper (FlowMapper.java:configure(72)) - child jvm opts: -Xmx200m
2015-02-12 16:12:47,712 INFO  hadoop.FlowMapper (FlowMapper.java:configure(87)) - sourcing from: Hfs["TextDelimited[['num', 'char']]"]["/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/classes/test/data/rhs.txt"]
2015-02-12 16:12:47,713 INFO  hadoop.FlowMapper (FlowMapper.java:configure(90)) - sinking to: GroupBy(count-lhs+count-rhs)[by: count-lhs:[{1}:'char'] count-rhs:[{1}:'char']]
2015-02-12 16:12:47,713 INFO  assembly.AggregateBy (AggregateBy.java:prepare(267)) - using threshold value: 2
2015-02-12 16:12:47,715 INFO  assembly.AggregateBy (AggregateBy.java:removeEldestEntry(327)) - flushed keys num times: 1, with threshold: 2
2015-02-12 16:12:47,715 INFO  assembly.AggregateBy (AggregateBy.java:removeEldestEntry(328)) - mem on flush (mb), free: 90, total: 502, max: 672
2015-02-12 16:12:47,715 INFO  assembly.AggregateBy (AggregateBy.java:removeEldestEntry(333)) - total mem is 74% of max mem, to better utilize unused memory consider increasing current LRU threshold with system property "cascading.aggregateby.threshold"
2015-02-12 16:12:47,716 INFO  assembly.AggregateBy (AggregateBy.java:removeEldestEntry(327)) - flushed keys num times: 3, with threshold: 2
2015-02-12 16:12:47,716 INFO  assembly.AggregateBy (AggregateBy.java:removeEldestEntry(328)) - mem on flush (mb), free: 90, total: 502, max: 672
2015-02-12 16:12:47,716 INFO  assembly.AggregateBy (AggregateBy.java:removeEldestEntry(333)) - total mem is 74% of max mem, to better utilize unused memory consider increasing current LRU threshold with system property "cascading.aggregateby.threshold"
2015-02-12 16:12:47,717 INFO  assembly.AggregateBy (AggregateBy.java:removeEldestEntry(327)) - flushed keys num times: 5, with threshold: 2
2015-02-12 16:12:47,717 INFO  assembly.AggregateBy (AggregateBy.java:removeEldestEntry(328)) - mem on flush (mb), free: 89, total: 502, max: 672
2015-02-12 16:12:47,717 INFO  assembly.AggregateBy (AggregateBy.java:removeEldestEntry(333)) - total mem is 74% of max mem, to better utilize unused memory consider increasing current LRU threshold with system property "cascading.aggregateby.threshold"
2015-02-12 16:12:47,717 INFO  assembly.AggregateBy (AggregateBy.java:removeEldestEntry(327)) - flushed keys num times: 7, with threshold: 2
2015-02-12 16:12:47,717 INFO  assembly.AggregateBy (AggregateBy.java:removeEldestEntry(328)) - mem on flush (mb), free: 89, total: 502, max: 672
2015-02-12 16:12:47,717 INFO  assembly.AggregateBy (AggregateBy.java:removeEldestEntry(333)) - total mem is 74% of max mem, to better utilize unused memory consider increasing current LRU threshold with system property "cascading.aggregateby.threshold"
2015-02-12 16:12:47,718 INFO  mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) - 
2015-02-12 16:12:47,718 INFO  mapred.MapTask (MapTask.java:flush(1457)) - Starting flush of map output
2015-02-12 16:12:47,718 INFO  mapred.MapTask (MapTask.java:flush(1475)) - Spilling map output
2015-02-12 16:12:47,718 INFO  mapred.MapTask (MapTask.java:flush(1476)) - bufstart = 0; bufend = 99; bufvoid = 104857600
2015-02-12 16:12:47,718 INFO  mapred.MapTask (MapTask.java:flush(1478)) - kvstart = 26214396(104857584); kvend = 26214364(104857456); length = 33/6553600
2015-02-12 16:12:47,720 INFO  mapred.MapTask (MapTask.java:sortAndSpill(1660)) - Finished spill 0
2015-02-12 16:12:47,722 INFO  mapred.Task (Task.java:done(1001)) - Task:attempt_local154647584_0007_m_000001_0 is done. And is in the process of committing
2015-02-12 16:12:47,723 INFO  mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) - file:/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/classes/test/data/rhs.txt:0+51
2015-02-12 16:12:47,723 INFO  mapred.Task (Task.java:sendDone(1121)) - Task 'attempt_local154647584_0007_m_000001_0' done.
2015-02-12 16:12:47,724 INFO  mapred.LocalJobRunner (LocalJobRunner.java:run(249)) - Finishing task: attempt_local154647584_0007_m_000001_0
2015-02-12 16:12:47,724 INFO  mapred.LocalJobRunner (LocalJobRunner.java:runTasks(456)) - map task executor complete.
2015-02-12 16:12:47,724 INFO  mapred.LocalJobRunner (LocalJobRunner.java:runTasks(448)) - Waiting for reduce tasks
2015-02-12 16:12:47,724 INFO  mapred.LocalJobRunner (LocalJobRunner.java:run(302)) - Starting task: attempt_local154647584_0007_r_000000_0
2015-02-12 16:12:47,726 INFO  util.ProcfsBasedProcessTree (ProcfsBasedProcessTree.java:isAvailable(181)) - ProcfsBasedProcessTree currently is supported only on Linux.
2015-02-12 16:12:47,726 INFO  mapred.Task (Task.java:initialize(587)) -  Using ResourceCalculatorProcessTree : null
2015-02-12 16:12:47,726 INFO  mapred.ReduceTask (ReduceTask.java:run(362)) - Using ShuffleConsumerPlugin: org.apache.hadoop.mapreduce.task.reduce.Shuffle@62aa34f5
2015-02-12 16:12:47,727 INFO  reduce.MergeManagerImpl (MergeManagerImpl.java:<init>(195)) - MergerManager: memoryLimit=493250144, maxSingleShuffleLimit=123312536, mergeThreshold=325545120, ioSortFactor=10, memToMemMergeOutputsThreshold=10
2015-02-12 16:12:47,727 INFO  reduce.EventFetcher (EventFetcher.java:run(61)) - attempt_local154647584_0007_r_000000_0 Thread started: EventFetcher for fetching Map Completion Events
2015-02-12 16:12:47,729 INFO  reduce.LocalFetcher (LocalFetcher.java:copyMapOutput(141)) - localfetcher#6 about to shuffle output of map attempt_local154647584_0007_m_000001_0 decomp: 119 len: 123 to MEMORY
2015-02-12 16:12:47,729 INFO  reduce.InMemoryMapOutput (InMemoryMapOutput.java:shuffle(100)) - Read 119 bytes from map-output for attempt_local154647584_0007_m_000001_0
2015-02-12 16:12:47,729 INFO  reduce.MergeManagerImpl (MergeManagerImpl.java:closeInMemoryFile(313)) - closeInMemoryFile -> map-output of size: 119, inMemoryMapOutputs.size() -> 1, commitMemory -> 0, usedMemory ->119
2015-02-12 16:12:47,731 INFO  reduce.LocalFetcher (LocalFetcher.java:copyMapOutput(141)) - localfetcher#6 about to shuffle output of map attempt_local154647584_0007_m_000000_0 decomp: 119 len: 123 to MEMORY
2015-02-12 16:12:47,732 INFO  reduce.InMemoryMapOutput (InMemoryMapOutput.java:shuffle(100)) - Read 119 bytes from map-output for attempt_local154647584_0007_m_000000_0
2015-02-12 16:12:47,732 INFO  reduce.MergeManagerImpl (MergeManagerImpl.java:closeInMemoryFile(313)) - closeInMemoryFile -> map-output of size: 119, inMemoryMapOutputs.size() -> 2, commitMemory -> 119, usedMemory ->238
2015-02-12 16:12:47,732 INFO  reduce.EventFetcher (EventFetcher.java:run(76)) - EventFetcher is interrupted.. Returning
2015-02-12 16:12:47,733 INFO  mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) - 2 / 2 copied.
2015-02-12 16:12:47,733 INFO  reduce.MergeManagerImpl (MergeManagerImpl.java:finalMerge(673)) - finalMerge called with 2 in-memory map-outputs and 0 on-disk map-outputs
2015-02-12 16:12:47,735 INFO  mapred.Merger (Merger.java:merge(606)) - Merging 2 sorted segments
2015-02-12 16:12:47,735 INFO  mapred.Merger (Merger.java:merge(705)) - Down to the last merge-pass, with 2 segments left of total size: 220 bytes
2015-02-12 16:12:47,737 INFO  reduce.MergeManagerImpl (MergeManagerImpl.java:finalMerge(750)) - Merged 2 segments, 238 bytes to disk to satisfy reduce memory limit
2015-02-12 16:12:47,737 INFO  reduce.MergeManagerImpl (MergeManagerImpl.java:finalMerge(780)) - Merging 1 files, 240 bytes from disk
2015-02-12 16:12:47,738 INFO  reduce.MergeManagerImpl (MergeManagerImpl.java:finalMerge(795)) - Merging 0 segments, 0 bytes from memory into reduce
2015-02-12 16:12:47,738 INFO  mapred.Merger (Merger.java:merge(606)) - Merging 1 sorted segments
2015-02-12 16:12:47,738 INFO  mapred.Merger (Merger.java:merge(705)) - Down to the last merge-pass, with 1 segments left of total size: 227 bytes
2015-02-12 16:12:47,739 INFO  mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) - 2 / 2 copied.
2015-02-12 16:12:47,739 INFO  hadoop.FlowReducer (FlowReducer.java:configure(78)) - cascading version: 2.6.3
2015-02-12 16:12:47,739 INFO  hadoop.FlowReducer (FlowReducer.java:configure(79)) - child jvm opts: -Xmx200m
2015-02-12 16:12:47,746 INFO  hadoop.FlowReducer (FlowReducer.java:configure(97)) - sourcing from: GroupBy(count-lhs+count-rhs)[by: count-lhs:[{1}:'char'] count-rhs:[{1}:'char']]
2015-02-12 16:12:47,746 INFO  hadoop.FlowReducer (FlowReducer.java:configure(100)) - sinking to: Hfs["TextDelimited[['char', 'count']]"]["/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/test/output/hadoop2-mr1/assemblyhelpersplatform/mergecount"]
2015-02-12 16:12:47,755 INFO  mapred.Task (Task.java:done(1001)) - Task:attempt_local154647584_0007_r_000000_0 is done. And is in the process of committing
2015-02-12 16:12:47,756 INFO  mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) - 2 / 2 copied.
2015-02-12 16:12:47,756 INFO  mapred.Task (Task.java:commit(1162)) - Task attempt_local154647584_0007_r_000000_0 is allowed to commit now
2015-02-12 16:12:47,757 INFO  output.FileOutputCommitter (FileOutputCommitter.java:commitTask(439)) - Saved output of task 'attempt_local154647584_0007_r_000000_0' to file:/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/test/output/hadoop2-mr1/assemblyhelpersplatform/mergecount/_temporary/0/task_local154647584_0007_r_000000
2015-02-12 16:12:47,758 INFO  mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) - reduce > reduce
2015-02-12 16:12:47,758 INFO  mapred.Task (Task.java:sendDone(1121)) - Task 'attempt_local154647584_0007_r_000000_0' done.
2015-02-12 16:12:47,758 INFO  mapred.LocalJobRunner (LocalJobRunner.java:run(325)) - Finishing task: attempt_local154647584_0007_r_000000_0
2015-02-12 16:12:47,758 INFO  mapred.LocalJobRunner (LocalJobRunner.java:runTasks(456)) - reduce task executor complete.
2015-02-12 16:12:47,784 INFO  util.Hadoop18TapUtil (Hadoop18TapUtil.java:cleanTempPath(219)) - deleting temp path /Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/test/output/hadoop2-mr1/assemblyhelpersplatform/mergecount/_temporary
2015-02-12 16:12:47,817 INFO  mapred.FileInputFormat (FileInputFormat.java:listStatus(247)) - Total input paths to process : 1
2015-02-12 16:12:47,856 INFO  mapred.FileInputFormat (FileInputFormat.java:listStatus(247)) - Total input paths to process : 1
2015-02-12 16:12:47,858 INFO  cascading.PlatformTestCase (PlatformTestCase.java:tearDown(209)) - copying to local /Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/test/output//hadoop2-mr1/assemblyhelpersplatform/mergecount
2015-02-12 16:12:47,879 INFO  util.HadoopUtil (HadoopUtil.java:findMainClass(336)) - using default application jar, may cause class not found exceptions on the cluster
2015-02-12 16:12:47,879 INFO  planner.HadoopPlanner (HadoopPlanner.java:initialize(225)) - using application jar: /Users/soberoi/.gradle/caches/modules-2/files-2.1/cascading/cascading-hadoop2-mr1/2.6.3/9ae016e2b8d246b9c1de9168e715d9f1b971aa4a/cascading-hadoop2-mr1-2.6.3.jar
2015-02-12 16:12:47,886 INFO  Configuration.deprecation (Configuration.java:warnOnceIfDeprecated(1022)) - mapred.used.genericoptionsparser is deprecated. Instead, use mapreduce.client.genericoptionsparser.used
2015-02-12 16:12:47,887 INFO  Configuration.deprecation (Configuration.java:warnOnceIfDeprecated(1022)) - mapred.job.tracker is deprecated. Instead, use mapreduce.jobtracker.address
2015-02-12 16:12:47,888 INFO  Configuration.deprecation (Configuration.java:warnOnceIfDeprecated(1022)) - mapred.output.compress is deprecated. Instead, use mapreduce.output.fileoutputformat.compress
2015-02-12 16:12:47,895 INFO  flow.Flow (BaseFlow.java:logInfo(1378)) - [min] starting
2015-02-12 16:12:47,895 INFO  flow.Flow (BaseFlow.java:logInfo(1378)) - [min]  source: Hfs["TextDelimited[['num', 'char']]"]["/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/classes/test/data/lhs.txt"]
2015-02-12 16:12:47,896 INFO  flow.Flow (BaseFlow.java:logInfo(1378)) - [min]  sink: Hfs["TextDelimited[['char', 'min']]"]["/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/test/output/hadoop2-mr1/assemblyhelpersplatform/minbynullsafety"]
2015-02-12 16:12:47,896 INFO  flow.Flow (BaseFlow.java:logInfo(1378)) - [min]  parallel execution is enabled: false
2015-02-12 16:12:47,896 INFO  flow.Flow (BaseFlow.java:logInfo(1378)) - [min]  starting jobs: 1
2015-02-12 16:12:47,896 INFO  flow.Flow (BaseFlow.java:logInfo(1378)) - [min]  allocating threads: 1
2015-02-12 16:12:47,896 INFO  flow.FlowStep (BaseFlowStep.java:logInfo(834)) - [min] starting step: (1/1) ...splatform/minbynullsafety
2015-02-12 16:12:47,897 INFO  jvm.JvmMetrics (JvmMetrics.java:init(71)) - Cannot initialize JVM Metrics with processName=JobTracker, sessionId= - already initialized
2015-02-12 16:12:47,898 INFO  jvm.JvmMetrics (JvmMetrics.java:init(71)) - Cannot initialize JVM Metrics with processName=JobTracker, sessionId= - already initialized
2015-02-12 16:12:47,918 INFO  mapred.FileInputFormat (FileInputFormat.java:listStatus(247)) - Total input paths to process : 1
2015-02-12 16:12:47,937 INFO  mapreduce.JobSubmitter (JobSubmitter.java:submitJobInternal(492)) - number of splits:1
2015-02-12 16:12:47,951 INFO  mapreduce.JobSubmitter (JobSubmitter.java:printTokens(575)) - Submitting tokens for job: job_local521802607_0008
2015-02-12 16:12:47,964 WARN  conf.Configuration (Configuration.java:loadProperty(2510)) - file:/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/tmp/cascading/staging/soberoi521802607/.staging/job_local521802607_0008/job.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.retry.interval;  Ignoring.
2015-02-12 16:12:47,965 WARN  conf.Configuration (Configuration.java:loadProperty(2510)) - file:/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/tmp/cascading/staging/soberoi521802607/.staging/job_local521802607_0008/job.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.attempts;  Ignoring.
2015-02-12 16:12:47,996 WARN  conf.Configuration (Configuration.java:loadProperty(2510)) - file:/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/test/tmp/mapred/local/localRunner/soberoi/job_local521802607_0008/job_local521802607_0008.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.retry.interval;  Ignoring.
2015-02-12 16:12:47,997 WARN  conf.Configuration (Configuration.java:loadProperty(2510)) - file:/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/test/tmp/mapred/local/localRunner/soberoi/job_local521802607_0008/job_local521802607_0008.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.attempts;  Ignoring.
2015-02-12 16:12:47,998 INFO  mapreduce.Job (Job.java:submit(1299)) - The url to track the job: http://localhost:8080/
2015-02-12 16:12:47,998 INFO  mapred.LocalJobRunner (LocalJobRunner.java:createOutputCommitter(471)) - OutputCommitter set in config null
2015-02-12 16:12:47,998 INFO  flow.FlowStep (BaseFlowStep.java:logInfo(834)) - [min] submitted hadoop job: job_local521802607_0008
2015-02-12 16:12:47,998 INFO  mapred.LocalJobRunner (LocalJobRunner.java:createOutputCommitter(489)) - OutputCommitter is org.apache.hadoop.mapred.FileOutputCommitter
2015-02-12 16:12:47,998 INFO  flow.FlowStep (BaseFlowStep.java:logInfo(834)) - [min] tracking url: http://localhost:8080/
2015-02-12 16:12:48,000 INFO  mapred.LocalJobRunner (LocalJobRunner.java:runTasks(448)) - Waiting for map tasks
2015-02-12 16:12:48,001 INFO  mapred.LocalJobRunner (LocalJobRunner.java:run(224)) - Starting task: attempt_local521802607_0008_m_000000_0
2015-02-12 16:12:48,002 INFO  util.ProcfsBasedProcessTree (ProcfsBasedProcessTree.java:isAvailable(181)) - ProcfsBasedProcessTree currently is supported only on Linux.
2015-02-12 16:12:48,002 INFO  mapred.Task (Task.java:initialize(587)) -  Using ResourceCalculatorProcessTree : null
2015-02-12 16:12:48,003 INFO  io.MultiInputSplit (MultiInputSplit.java:readFields(161)) - current split input path: file:/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/classes/test/data/lhs.txt
2015-02-12 16:12:48,003 INFO  mapred.MapTask (MapTask.java:updateJobWithSplit(482)) - Processing split: cascading.tap.hadoop.io.MultiInputSplit@64cd6b49
2015-02-12 16:12:48,004 INFO  mapred.MapTask (MapTask.java:runOldMapper(436)) - numReduceTasks: 1
2015-02-12 16:12:48,015 INFO  mapred.MapTask (MapTask.java:setEquator(1202)) - (EQUATOR) 0 kvi 26214396(104857584)
2015-02-12 16:12:48,015 INFO  mapred.MapTask (MapTask.java:init(995)) - mapreduce.task.io.sort.mb: 100
2015-02-12 16:12:48,015 INFO  mapred.MapTask (MapTask.java:init(996)) - soft limit at 83886080
2015-02-12 16:12:48,016 INFO  mapred.MapTask (MapTask.java:init(997)) - bufstart = 0; bufvoid = 104857600
2015-02-12 16:12:48,016 INFO  mapred.MapTask (MapTask.java:init(998)) - kvstart = 26214396; length = 6553600
2015-02-12 16:12:48,016 INFO  mapred.MapTask (MapTask.java:createSortingCollector(402)) - Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer
2015-02-12 16:12:48,016 INFO  hadoop.FlowMapper (FlowMapper.java:configure(71)) - cascading version: 2.6.3
2015-02-12 16:12:48,017 INFO  hadoop.FlowMapper (FlowMapper.java:configure(72)) - child jvm opts: -Xmx200m
2015-02-12 16:12:48,022 INFO  hadoop.FlowMapper (FlowMapper.java:configure(87)) - sourcing from: Hfs["TextDelimited[['num', 'char']]"]["/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/classes/test/data/lhs.txt"]
2015-02-12 16:12:48,022 INFO  hadoop.FlowMapper (FlowMapper.java:configure(90)) - sinking to: GroupBy(min)[by:[{1}:'char']]
2015-02-12 16:12:48,023 INFO  assembly.AggregateBy (AggregateBy.java:prepare(267)) - using threshold value: 2
2015-02-12 16:12:48,024 INFO  assembly.AggregateBy (AggregateBy.java:removeEldestEntry(327)) - flushed keys num times: 1, with threshold: 2
2015-02-12 16:12:48,024 INFO  assembly.AggregateBy (AggregateBy.java:removeEldestEntry(328)) - mem on flush (mb), free: 158, total: 509, max: 672
2015-02-12 16:12:48,024 INFO  assembly.AggregateBy (AggregateBy.java:removeEldestEntry(333)) - total mem is 75% of max mem, to better utilize unused memory consider increasing current LRU threshold with system property "cascading.aggregateby.threshold"
2015-02-12 16:12:48,024 INFO  assembly.AggregateBy (AggregateBy.java:removeEldestEntry(327)) - flushed keys num times: 3, with threshold: 2
2015-02-12 16:12:48,025 INFO  assembly.AggregateBy (AggregateBy.java:removeEldestEntry(328)) - mem on flush (mb), free: 158, total: 509, max: 672
2015-02-12 16:12:48,025 INFO  assembly.AggregateBy (AggregateBy.java:removeEldestEntry(333)) - total mem is 75% of max mem, to better utilize unused memory consider increasing current LRU threshold with system property "cascading.aggregateby.threshold"
2015-02-12 16:12:48,025 INFO  assembly.AggregateBy (AggregateBy.java:removeEldestEntry(327)) - flushed keys num times: 5, with threshold: 2
2015-02-12 16:12:48,025 INFO  assembly.AggregateBy (AggregateBy.java:removeEldestEntry(328)) - mem on flush (mb), free: 158, total: 509, max: 672
2015-02-12 16:12:48,025 INFO  assembly.AggregateBy (AggregateBy.java:removeEldestEntry(333)) - total mem is 75% of max mem, to better utilize unused memory consider increasing current LRU threshold with system property "cascading.aggregateby.threshold"
2015-02-12 16:12:48,026 INFO  assembly.AggregateBy (AggregateBy.java:removeEldestEntry(327)) - flushed keys num times: 7, with threshold: 2
2015-02-12 16:12:48,026 INFO  assembly.AggregateBy (AggregateBy.java:removeEldestEntry(328)) - mem on flush (mb), free: 158, total: 509, max: 672
2015-02-12 16:12:48,026 INFO  assembly.AggregateBy (AggregateBy.java:removeEldestEntry(333)) - total mem is 75% of max mem, to better utilize unused memory consider increasing current LRU threshold with system property "cascading.aggregateby.threshold"
2015-02-12 16:12:48,027 INFO  mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) - 
2015-02-12 16:12:48,027 INFO  mapred.MapTask (MapTask.java:flush(1457)) - Starting flush of map output
2015-02-12 16:12:48,027 INFO  mapred.MapTask (MapTask.java:flush(1475)) - Spilling map output
2015-02-12 16:12:48,027 INFO  mapred.MapTask (MapTask.java:flush(1476)) - bufstart = 0; bufend = 99; bufvoid = 104857600
2015-02-12 16:12:48,027 INFO  mapred.MapTask (MapTask.java:flush(1478)) - kvstart = 26214396(104857584); kvend = 26214364(104857456); length = 33/6553600
2015-02-12 16:12:48,030 INFO  mapred.MapTask (MapTask.java:sortAndSpill(1660)) - Finished spill 0
2015-02-12 16:12:48,032 INFO  mapred.Task (Task.java:done(1001)) - Task:attempt_local521802607_0008_m_000000_0 is done. And is in the process of committing
2015-02-12 16:12:48,034 INFO  mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) - file:/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/classes/test/data/lhs.txt:0+52
2015-02-12 16:12:48,034 INFO  mapred.Task (Task.java:sendDone(1121)) - Task 'attempt_local521802607_0008_m_000000_0' done.
2015-02-12 16:12:48,034 INFO  mapred.LocalJobRunner (LocalJobRunner.java:run(249)) - Finishing task: attempt_local521802607_0008_m_000000_0
2015-02-12 16:12:48,034 INFO  mapred.LocalJobRunner (LocalJobRunner.java:runTasks(456)) - map task executor complete.
2015-02-12 16:12:48,035 INFO  mapred.LocalJobRunner (LocalJobRunner.java:runTasks(448)) - Waiting for reduce tasks
2015-02-12 16:12:48,035 INFO  mapred.LocalJobRunner (LocalJobRunner.java:run(302)) - Starting task: attempt_local521802607_0008_r_000000_0
2015-02-12 16:12:48,036 INFO  util.ProcfsBasedProcessTree (ProcfsBasedProcessTree.java:isAvailable(181)) - ProcfsBasedProcessTree currently is supported only on Linux.
2015-02-12 16:12:48,036 INFO  mapred.Task (Task.java:initialize(587)) -  Using ResourceCalculatorProcessTree : null
2015-02-12 16:12:48,036 INFO  mapred.ReduceTask (ReduceTask.java:run(362)) - Using ShuffleConsumerPlugin: org.apache.hadoop.mapreduce.task.reduce.Shuffle@263d40c
2015-02-12 16:12:48,037 INFO  reduce.MergeManagerImpl (MergeManagerImpl.java:<init>(195)) - MergerManager: memoryLimit=493250144, maxSingleShuffleLimit=123312536, mergeThreshold=325545120, ioSortFactor=10, memToMemMergeOutputsThreshold=10
2015-02-12 16:12:48,037 INFO  reduce.EventFetcher (EventFetcher.java:run(61)) - attempt_local521802607_0008_r_000000_0 Thread started: EventFetcher for fetching Map Completion Events
2015-02-12 16:12:48,040 INFO  reduce.LocalFetcher (LocalFetcher.java:copyMapOutput(141)) - localfetcher#7 about to shuffle output of map attempt_local521802607_0008_m_000000_0 decomp: 119 len: 123 to MEMORY
2015-02-12 16:12:48,041 INFO  reduce.InMemoryMapOutput (InMemoryMapOutput.java:shuffle(100)) - Read 119 bytes from map-output for attempt_local521802607_0008_m_000000_0
2015-02-12 16:12:48,041 INFO  reduce.MergeManagerImpl (MergeManagerImpl.java:closeInMemoryFile(313)) - closeInMemoryFile -> map-output of size: 119, inMemoryMapOutputs.size() -> 1, commitMemory -> 0, usedMemory ->119
2015-02-12 16:12:48,042 INFO  reduce.EventFetcher (EventFetcher.java:run(76)) - EventFetcher is interrupted.. Returning
2015-02-12 16:12:48,042 INFO  mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) - 1 / 1 copied.
2015-02-12 16:12:48,042 INFO  reduce.MergeManagerImpl (MergeManagerImpl.java:finalMerge(673)) - finalMerge called with 1 in-memory map-outputs and 0 on-disk map-outputs
2015-02-12 16:12:48,044 INFO  mapred.Merger (Merger.java:merge(606)) - Merging 1 sorted segments
2015-02-12 16:12:48,044 INFO  mapred.Merger (Merger.java:merge(705)) - Down to the last merge-pass, with 1 segments left of total size: 110 bytes
2015-02-12 16:12:48,046 INFO  reduce.MergeManagerImpl (MergeManagerImpl.java:finalMerge(750)) - Merged 1 segments, 119 bytes to disk to satisfy reduce memory limit
2015-02-12 16:12:48,046 INFO  reduce.MergeManagerImpl (MergeManagerImpl.java:finalMerge(780)) - Merging 1 files, 123 bytes from disk
2015-02-12 16:12:48,046 INFO  reduce.MergeManagerImpl (MergeManagerImpl.java:finalMerge(795)) - Merging 0 segments, 0 bytes from memory into reduce
2015-02-12 16:12:48,046 INFO  mapred.Merger (Merger.java:merge(606)) - Merging 1 sorted segments
2015-02-12 16:12:48,047 INFO  mapred.Merger (Merger.java:merge(705)) - Down to the last merge-pass, with 1 segments left of total size: 110 bytes
2015-02-12 16:12:48,047 INFO  mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) - 1 / 1 copied.
2015-02-12 16:12:48,047 INFO  hadoop.FlowReducer (FlowReducer.java:configure(78)) - cascading version: 2.6.3
2015-02-12 16:12:48,047 INFO  hadoop.FlowReducer (FlowReducer.java:configure(79)) - child jvm opts: -Xmx200m
2015-02-12 16:12:48,055 INFO  hadoop.FlowReducer (FlowReducer.java:configure(97)) - sourcing from: GroupBy(min)[by:[{1}:'char']]
2015-02-12 16:12:48,055 INFO  hadoop.FlowReducer (FlowReducer.java:configure(100)) - sinking to: Hfs["TextDelimited[['char', 'min']]"]["/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/test/output/hadoop2-mr1/assemblyhelpersplatform/minbynullsafety"]
2015-02-12 16:12:48,065 INFO  mapred.Task (Task.java:done(1001)) - Task:attempt_local521802607_0008_r_000000_0 is done. And is in the process of committing
2015-02-12 16:12:48,066 INFO  mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) - 1 / 1 copied.
2015-02-12 16:12:48,066 INFO  mapred.Task (Task.java:commit(1162)) - Task attempt_local521802607_0008_r_000000_0 is allowed to commit now
2015-02-12 16:12:48,067 INFO  output.FileOutputCommitter (FileOutputCommitter.java:commitTask(439)) - Saved output of task 'attempt_local521802607_0008_r_000000_0' to file:/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/test/output/hadoop2-mr1/assemblyhelpersplatform/minbynullsafety/_temporary/0/task_local521802607_0008_r_000000
2015-02-12 16:12:48,068 INFO  mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) - reduce > reduce
2015-02-12 16:12:48,068 INFO  mapred.Task (Task.java:sendDone(1121)) - Task 'attempt_local521802607_0008_r_000000_0' done.
2015-02-12 16:12:48,068 INFO  mapred.LocalJobRunner (LocalJobRunner.java:run(325)) - Finishing task: attempt_local521802607_0008_r_000000_0
2015-02-12 16:12:48,068 INFO  mapred.LocalJobRunner (LocalJobRunner.java:runTasks(456)) - reduce task executor complete.
2015-02-12 16:12:48,085 INFO  util.Hadoop18TapUtil (Hadoop18TapUtil.java:cleanTempPath(219)) - deleting temp path /Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/test/output/hadoop2-mr1/assemblyhelpersplatform/minbynullsafety/_temporary
2015-02-12 16:12:48,107 INFO  mapred.FileInputFormat (FileInputFormat.java:listStatus(247)) - Total input paths to process : 1
2015-02-12 16:12:48,139 INFO  mapred.FileInputFormat (FileInputFormat.java:listStatus(247)) - Total input paths to process : 1
2015-02-12 16:12:48,141 INFO  cascading.PlatformTestCase (PlatformTestCase.java:tearDown(209)) - copying to local /Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/test/output//hadoop2-mr1/assemblyhelpersplatform/minbynullsafety
2015-02-12 16:12:48,160 INFO  util.HadoopUtil (HadoopUtil.java:findMainClass(336)) - using default application jar, may cause class not found exceptions on the cluster
2015-02-12 16:12:48,161 INFO  planner.HadoopPlanner (HadoopPlanner.java:initialize(225)) - using application jar: /Users/soberoi/.gradle/caches/modules-2/files-2.1/cascading/cascading-hadoop2-mr1/2.6.3/9ae016e2b8d246b9c1de9168e715d9f1b971aa4a/cascading-hadoop2-mr1-2.6.3.jar
2015-02-12 16:12:48,166 INFO  Configuration.deprecation (Configuration.java:warnOnceIfDeprecated(1022)) - mapred.used.genericoptionsparser is deprecated. Instead, use mapreduce.client.genericoptionsparser.used
2015-02-12 16:12:48,168 INFO  Configuration.deprecation (Configuration.java:warnOnceIfDeprecated(1022)) - mapred.job.tracker is deprecated. Instead, use mapreduce.jobtracker.address
2015-02-12 16:12:48,168 INFO  Configuration.deprecation (Configuration.java:warnOnceIfDeprecated(1022)) - mapred.output.compress is deprecated. Instead, use mapreduce.output.fileoutputformat.compress
2015-02-12 16:12:48,178 INFO  flow.Flow (BaseFlow.java:logInfo(1378)) - [max] starting
2015-02-12 16:12:48,179 INFO  flow.Flow (BaseFlow.java:logInfo(1378)) - [max]  source: Hfs["TextDelimited[['num', 'char']]"]["/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/classes/test/data/lhs.txt"]
2015-02-12 16:12:48,179 INFO  flow.Flow (BaseFlow.java:logInfo(1378)) - [max]  sink: Hfs["TextDelimited[['num', 'max']]"]["/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/test/output/hadoop2-mr1/assemblyhelpersplatform/maxbystring"]
2015-02-12 16:12:48,179 INFO  flow.Flow (BaseFlow.java:logInfo(1378)) - [max]  parallel execution is enabled: false
2015-02-12 16:12:48,179 INFO  flow.Flow (BaseFlow.java:logInfo(1378)) - [max]  starting jobs: 1
2015-02-12 16:12:48,179 INFO  flow.Flow (BaseFlow.java:logInfo(1378)) - [max]  allocating threads: 1
2015-02-12 16:12:48,180 INFO  flow.FlowStep (BaseFlowStep.java:logInfo(834)) - [max] starting step: (1/1) ...lpersplatform/maxbystring
2015-02-12 16:12:48,181 INFO  jvm.JvmMetrics (JvmMetrics.java:init(71)) - Cannot initialize JVM Metrics with processName=JobTracker, sessionId= - already initialized
2015-02-12 16:12:48,182 INFO  jvm.JvmMetrics (JvmMetrics.java:init(71)) - Cannot initialize JVM Metrics with processName=JobTracker, sessionId= - already initialized
2015-02-12 16:12:48,211 INFO  mapred.FileInputFormat (FileInputFormat.java:listStatus(247)) - Total input paths to process : 1
2015-02-12 16:12:48,236 INFO  mapreduce.JobSubmitter (JobSubmitter.java:submitJobInternal(492)) - number of splits:1
2015-02-12 16:12:48,253 INFO  mapreduce.JobSubmitter (JobSubmitter.java:printTokens(575)) - Submitting tokens for job: job_local654358189_0009
2015-02-12 16:12:48,273 WARN  conf.Configuration (Configuration.java:loadProperty(2510)) - file:/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/tmp/cascading/staging/soberoi654358189/.staging/job_local654358189_0009/job.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.retry.interval;  Ignoring.
2015-02-12 16:12:48,274 WARN  conf.Configuration (Configuration.java:loadProperty(2510)) - file:/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/tmp/cascading/staging/soberoi654358189/.staging/job_local654358189_0009/job.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.attempts;  Ignoring.
2015-02-12 16:12:48,316 WARN  conf.Configuration (Configuration.java:loadProperty(2510)) - file:/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/test/tmp/mapred/local/localRunner/soberoi/job_local654358189_0009/job_local654358189_0009.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.retry.interval;  Ignoring.
2015-02-12 16:12:48,317 WARN  conf.Configuration (Configuration.java:loadProperty(2510)) - file:/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/test/tmp/mapred/local/localRunner/soberoi/job_local654358189_0009/job_local654358189_0009.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.attempts;  Ignoring.
2015-02-12 16:12:48,318 INFO  mapreduce.Job (Job.java:submit(1299)) - The url to track the job: http://localhost:8080/
2015-02-12 16:12:48,318 INFO  mapred.LocalJobRunner (LocalJobRunner.java:createOutputCommitter(471)) - OutputCommitter set in config null
2015-02-12 16:12:48,318 INFO  flow.FlowStep (BaseFlowStep.java:logInfo(834)) - [max] submitted hadoop job: job_local654358189_0009
2015-02-12 16:12:48,318 INFO  mapred.LocalJobRunner (LocalJobRunner.java:createOutputCommitter(489)) - OutputCommitter is org.apache.hadoop.mapred.FileOutputCommitter
2015-02-12 16:12:48,318 INFO  flow.FlowStep (BaseFlowStep.java:logInfo(834)) - [max] tracking url: http://localhost:8080/
2015-02-12 16:12:48,321 INFO  mapred.LocalJobRunner (LocalJobRunner.java:runTasks(448)) - Waiting for map tasks
2015-02-12 16:12:48,321 INFO  mapred.LocalJobRunner (LocalJobRunner.java:run(224)) - Starting task: attempt_local654358189_0009_m_000000_0
2015-02-12 16:12:48,322 INFO  util.ProcfsBasedProcessTree (ProcfsBasedProcessTree.java:isAvailable(181)) - ProcfsBasedProcessTree currently is supported only on Linux.
2015-02-12 16:12:48,322 INFO  mapred.Task (Task.java:initialize(587)) -  Using ResourceCalculatorProcessTree : null
2015-02-12 16:12:48,323 INFO  io.MultiInputSplit (MultiInputSplit.java:readFields(161)) - current split input path: file:/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/classes/test/data/lhs.txt
2015-02-12 16:12:48,324 INFO  mapred.MapTask (MapTask.java:updateJobWithSplit(482)) - Processing split: cascading.tap.hadoop.io.MultiInputSplit@4e5a67df
2015-02-12 16:12:48,325 INFO  mapred.MapTask (MapTask.java:runOldMapper(436)) - numReduceTasks: 1
2015-02-12 16:12:48,341 INFO  mapred.MapTask (MapTask.java:setEquator(1202)) - (EQUATOR) 0 kvi 26214396(104857584)
2015-02-12 16:12:48,341 INFO  mapred.MapTask (MapTask.java:init(995)) - mapreduce.task.io.sort.mb: 100
2015-02-12 16:12:48,341 INFO  mapred.MapTask (MapTask.java:init(996)) - soft limit at 83886080
2015-02-12 16:12:48,342 INFO  mapred.MapTask (MapTask.java:init(997)) - bufstart = 0; bufvoid = 104857600
2015-02-12 16:12:48,342 INFO  mapred.MapTask (MapTask.java:init(998)) - kvstart = 26214396; length = 6553600
2015-02-12 16:12:48,343 INFO  mapred.MapTask (MapTask.java:createSortingCollector(402)) - Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer
2015-02-12 16:12:48,343 INFO  hadoop.FlowMapper (FlowMapper.java:configure(71)) - cascading version: 2.6.3
2015-02-12 16:12:48,343 INFO  hadoop.FlowMapper (FlowMapper.java:configure(72)) - child jvm opts: -Xmx200m
2015-02-12 16:12:48,348 INFO  hadoop.FlowMapper (FlowMapper.java:configure(87)) - sourcing from: Hfs["TextDelimited[['num', 'char']]"]["/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/classes/test/data/lhs.txt"]
2015-02-12 16:12:48,348 INFO  hadoop.FlowMapper (FlowMapper.java:configure(90)) - sinking to: GroupBy(max)[by:[{1}:'num']]
2015-02-12 16:12:48,348 INFO  assembly.AggregateBy (AggregateBy.java:prepare(267)) - using threshold value: 2
2015-02-12 16:12:48,349 INFO  assembly.AggregateBy (AggregateBy.java:removeEldestEntry(327)) - flushed keys num times: 1, with threshold: 2
2015-02-12 16:12:48,350 INFO  assembly.AggregateBy (AggregateBy.java:removeEldestEntry(328)) - mem on flush (mb), free: 189, total: 499, max: 672
2015-02-12 16:12:48,350 INFO  assembly.AggregateBy (AggregateBy.java:removeEldestEntry(333)) - total mem is 74% of max mem, to better utilize unused memory consider increasing current LRU threshold with system property "cascading.aggregateby.threshold"
2015-02-12 16:12:48,350 INFO  assembly.AggregateBy (AggregateBy.java:removeEldestEntry(327)) - flushed keys num times: 3, with threshold: 2
2015-02-12 16:12:48,350 INFO  assembly.AggregateBy (AggregateBy.java:removeEldestEntry(328)) - mem on flush (mb), free: 189, total: 499, max: 672
2015-02-12 16:12:48,350 INFO  assembly.AggregateBy (AggregateBy.java:removeEldestEntry(333)) - total mem is 74% of max mem, to better utilize unused memory consider increasing current LRU threshold with system property "cascading.aggregateby.threshold"
2015-02-12 16:12:48,352 INFO  mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) - 
2015-02-12 16:12:48,352 INFO  mapred.MapTask (MapTask.java:flush(1457)) - Starting flush of map output
2015-02-12 16:12:48,352 INFO  mapred.MapTask (MapTask.java:flush(1475)) - Spilling map output
2015-02-12 16:12:48,352 INFO  mapred.MapTask (MapTask.java:flush(1476)) - bufstart = 0; bufend = 75; bufvoid = 104857600
2015-02-12 16:12:48,352 INFO  mapred.MapTask (MapTask.java:flush(1478)) - kvstart = 26214396(104857584); kvend = 26214380(104857520); length = 17/6553600
2015-02-12 16:12:48,354 INFO  mapred.MapTask (MapTask.java:sortAndSpill(1660)) - Finished spill 0
2015-02-12 16:12:48,355 INFO  mapred.Task (Task.java:done(1001)) - Task:attempt_local654358189_0009_m_000000_0 is done. And is in the process of committing
2015-02-12 16:12:48,356 INFO  mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) - file:/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/classes/test/data/lhs.txt:0+52
2015-02-12 16:12:48,356 INFO  mapred.Task (Task.java:sendDone(1121)) - Task 'attempt_local654358189_0009_m_000000_0' done.
2015-02-12 16:12:48,356 INFO  mapred.LocalJobRunner (LocalJobRunner.java:run(249)) - Finishing task: attempt_local654358189_0009_m_000000_0
2015-02-12 16:12:48,356 INFO  mapred.LocalJobRunner (LocalJobRunner.java:runTasks(456)) - map task executor complete.
2015-02-12 16:12:48,357 INFO  mapred.LocalJobRunner (LocalJobRunner.java:runTasks(448)) - Waiting for reduce tasks
2015-02-12 16:12:48,357 INFO  mapred.LocalJobRunner (LocalJobRunner.java:run(302)) - Starting task: attempt_local654358189_0009_r_000000_0
2015-02-12 16:12:48,358 INFO  util.ProcfsBasedProcessTree (ProcfsBasedProcessTree.java:isAvailable(181)) - ProcfsBasedProcessTree currently is supported only on Linux.
2015-02-12 16:12:48,358 INFO  mapred.Task (Task.java:initialize(587)) -  Using ResourceCalculatorProcessTree : null
2015-02-12 16:12:48,358 INFO  mapred.ReduceTask (ReduceTask.java:run(362)) - Using ShuffleConsumerPlugin: org.apache.hadoop.mapreduce.task.reduce.Shuffle@45b9d56f
2015-02-12 16:12:48,359 INFO  reduce.MergeManagerImpl (MergeManagerImpl.java:<init>(195)) - MergerManager: memoryLimit=493250144, maxSingleShuffleLimit=123312536, mergeThreshold=325545120, ioSortFactor=10, memToMemMergeOutputsThreshold=10
2015-02-12 16:12:48,359 INFO  reduce.EventFetcher (EventFetcher.java:run(61)) - attempt_local654358189_0009_r_000000_0 Thread started: EventFetcher for fetching Map Completion Events
2015-02-12 16:12:48,361 INFO  reduce.LocalFetcher (LocalFetcher.java:copyMapOutput(141)) - localfetcher#8 about to shuffle output of map attempt_local654358189_0009_m_000000_0 decomp: 87 len: 91 to MEMORY
2015-02-12 16:12:48,361 INFO  reduce.InMemoryMapOutput (InMemoryMapOutput.java:shuffle(100)) - Read 87 bytes from map-output for attempt_local654358189_0009_m_000000_0
2015-02-12 16:12:48,361 INFO  reduce.MergeManagerImpl (MergeManagerImpl.java:closeInMemoryFile(313)) - closeInMemoryFile -> map-output of size: 87, inMemoryMapOutputs.size() -> 1, commitMemory -> 0, usedMemory ->87
2015-02-12 16:12:48,362 INFO  reduce.EventFetcher (EventFetcher.java:run(76)) - EventFetcher is interrupted.. Returning
2015-02-12 16:12:48,362 INFO  mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) - 1 / 1 copied.
2015-02-12 16:12:48,362 INFO  reduce.MergeManagerImpl (MergeManagerImpl.java:finalMerge(673)) - finalMerge called with 1 in-memory map-outputs and 0 on-disk map-outputs
2015-02-12 16:12:48,364 INFO  mapred.Merger (Merger.java:merge(606)) - Merging 1 sorted segments
2015-02-12 16:12:48,364 INFO  mapred.Merger (Merger.java:merge(705)) - Down to the last merge-pass, with 1 segments left of total size: 78 bytes
2015-02-12 16:12:48,365 INFO  reduce.MergeManagerImpl (MergeManagerImpl.java:finalMerge(750)) - Merged 1 segments, 87 bytes to disk to satisfy reduce memory limit
2015-02-12 16:12:48,365 INFO  reduce.MergeManagerImpl (MergeManagerImpl.java:finalMerge(780)) - Merging 1 files, 91 bytes from disk
2015-02-12 16:12:48,366 INFO  reduce.MergeManagerImpl (MergeManagerImpl.java:finalMerge(795)) - Merging 0 segments, 0 bytes from memory into reduce
2015-02-12 16:12:48,366 INFO  mapred.Merger (Merger.java:merge(606)) - Merging 1 sorted segments
2015-02-12 16:12:48,366 INFO  mapred.Merger (Merger.java:merge(705)) - Down to the last merge-pass, with 1 segments left of total size: 78 bytes
2015-02-12 16:12:48,366 INFO  mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) - 1 / 1 copied.
2015-02-12 16:12:48,367 INFO  hadoop.FlowReducer (FlowReducer.java:configure(78)) - cascading version: 2.6.3
2015-02-12 16:12:48,367 INFO  hadoop.FlowReducer (FlowReducer.java:configure(79)) - child jvm opts: -Xmx200m
2015-02-12 16:12:48,373 INFO  hadoop.FlowReducer (FlowReducer.java:configure(97)) - sourcing from: GroupBy(max)[by:[{1}:'num']]
2015-02-12 16:12:48,374 INFO  hadoop.FlowReducer (FlowReducer.java:configure(100)) - sinking to: Hfs["TextDelimited[['num', 'max']]"]["/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/test/output/hadoop2-mr1/assemblyhelpersplatform/maxbystring"]
2015-02-12 16:12:48,385 INFO  mapred.Task (Task.java:done(1001)) - Task:attempt_local654358189_0009_r_000000_0 is done. And is in the process of committing
2015-02-12 16:12:48,386 INFO  mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) - 1 / 1 copied.
2015-02-12 16:12:48,386 INFO  mapred.Task (Task.java:commit(1162)) - Task attempt_local654358189_0009_r_000000_0 is allowed to commit now
2015-02-12 16:12:48,387 INFO  output.FileOutputCommitter (FileOutputCommitter.java:commitTask(439)) - Saved output of task 'attempt_local654358189_0009_r_000000_0' to file:/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/test/output/hadoop2-mr1/assemblyhelpersplatform/maxbystring/_temporary/0/task_local654358189_0009_r_000000
2015-02-12 16:12:48,388 INFO  mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) - reduce > reduce
2015-02-12 16:12:48,388 INFO  mapred.Task (Task.java:sendDone(1121)) - Task 'attempt_local654358189_0009_r_000000_0' done.
2015-02-12 16:12:48,388 INFO  mapred.LocalJobRunner (LocalJobRunner.java:run(325)) - Finishing task: attempt_local654358189_0009_r_000000_0
2015-02-12 16:12:48,388 INFO  mapred.LocalJobRunner (LocalJobRunner.java:runTasks(456)) - reduce task executor complete.
2015-02-12 16:12:48,408 INFO  util.Hadoop18TapUtil (Hadoop18TapUtil.java:cleanTempPath(219)) - deleting temp path /Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/test/output/hadoop2-mr1/assemblyhelpersplatform/maxbystring/_temporary
2015-02-12 16:12:48,433 INFO  mapred.FileInputFormat (FileInputFormat.java:listStatus(247)) - Total input paths to process : 1
2015-02-12 16:12:48,455 INFO  mapred.FileInputFormat (FileInputFormat.java:listStatus(247)) - Total input paths to process : 1
2015-02-12 16:12:48,457 INFO  cascading.PlatformTestCase (PlatformTestCase.java:tearDown(209)) - copying to local /Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/test/output//hadoop2-mr1/assemblyhelpersplatform/maxbystring
2015-02-12 16:12:48,474 INFO  util.HadoopUtil (HadoopUtil.java:findMainClass(336)) - using default application jar, may cause class not found exceptions on the cluster
2015-02-12 16:12:48,475 INFO  planner.HadoopPlanner (HadoopPlanner.java:initialize(225)) - using application jar: /Users/soberoi/.gradle/caches/modules-2/files-2.1/cascading/cascading-hadoop2-mr1/2.6.3/9ae016e2b8d246b9c1de9168e715d9f1b971aa4a/cascading-hadoop2-mr1-2.6.3.jar
2015-02-12 16:12:48,482 INFO  Configuration.deprecation (Configuration.java:warnOnceIfDeprecated(1022)) - mapred.used.genericoptionsparser is deprecated. Instead, use mapreduce.client.genericoptionsparser.used
2015-02-12 16:12:48,484 INFO  Configuration.deprecation (Configuration.java:warnOnceIfDeprecated(1022)) - mapred.job.tracker is deprecated. Instead, use mapreduce.jobtracker.address
2015-02-12 16:12:48,485 INFO  Configuration.deprecation (Configuration.java:warnOnceIfDeprecated(1022)) - mapred.output.compress is deprecated. Instead, use mapreduce.output.fileoutputformat.compress
2015-02-12 16:12:48,496 INFO  flow.Flow (BaseFlow.java:logInfo(1378)) - [first] starting
2015-02-12 16:12:48,497 INFO  flow.Flow (BaseFlow.java:logInfo(1378)) - [first]  source: Hfs["TextDelimited[['num', 'lower', 'upper']]"]["/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/classes/test/data/lhs+rhs-cross.txt"]
2015-02-12 16:12:48,497 INFO  flow.Flow (BaseFlow.java:logInfo(1378)) - [first]  sink: Hfs["TextDelimited[['num', 'lower', 'upper']]"]["/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/test/output/hadoop2-mr1/assemblyhelpersplatform/firstnfields"]
2015-02-12 16:12:48,497 INFO  flow.Flow (BaseFlow.java:logInfo(1378)) - [first]  parallel execution is enabled: false
2015-02-12 16:12:48,497 INFO  flow.Flow (BaseFlow.java:logInfo(1378)) - [first]  starting jobs: 1
2015-02-12 16:12:48,497 INFO  flow.Flow (BaseFlow.java:logInfo(1378)) - [first]  allocating threads: 1
2015-02-12 16:12:48,497 INFO  flow.FlowStep (BaseFlowStep.java:logInfo(834)) - [first] starting step: (1/1) ...persplatform/firstnfields
2015-02-12 16:12:48,498 INFO  jvm.JvmMetrics (JvmMetrics.java:init(71)) - Cannot initialize JVM Metrics with processName=JobTracker, sessionId= - already initialized
2015-02-12 16:12:48,498 INFO  jvm.JvmMetrics (JvmMetrics.java:init(71)) - Cannot initialize JVM Metrics with processName=JobTracker, sessionId= - already initialized
2015-02-12 16:12:48,516 INFO  mapred.FileInputFormat (FileInputFormat.java:listStatus(247)) - Total input paths to process : 1
2015-02-12 16:12:48,532 INFO  mapreduce.JobSubmitter (JobSubmitter.java:submitJobInternal(492)) - number of splits:1
2015-02-12 16:12:48,543 INFO  mapreduce.JobSubmitter (JobSubmitter.java:printTokens(575)) - Submitting tokens for job: job_local2080117784_0010
2015-02-12 16:12:48,553 WARN  conf.Configuration (Configuration.java:loadProperty(2510)) - file:/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/tmp/cascading/staging/soberoi2080117784/.staging/job_local2080117784_0010/job.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.retry.interval;  Ignoring.
2015-02-12 16:12:48,554 WARN  conf.Configuration (Configuration.java:loadProperty(2510)) - file:/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/tmp/cascading/staging/soberoi2080117784/.staging/job_local2080117784_0010/job.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.attempts;  Ignoring.
2015-02-12 16:12:48,589 WARN  conf.Configuration (Configuration.java:loadProperty(2510)) - file:/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/test/tmp/mapred/local/localRunner/soberoi/job_local2080117784_0010/job_local2080117784_0010.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.retry.interval;  Ignoring.
2015-02-12 16:12:48,590 WARN  conf.Configuration (Configuration.java:loadProperty(2510)) - file:/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/test/tmp/mapred/local/localRunner/soberoi/job_local2080117784_0010/job_local2080117784_0010.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.attempts;  Ignoring.
2015-02-12 16:12:48,591 INFO  mapreduce.Job (Job.java:submit(1299)) - The url to track the job: http://localhost:8080/
2015-02-12 16:12:48,591 INFO  mapred.LocalJobRunner (LocalJobRunner.java:createOutputCommitter(471)) - OutputCommitter set in config null
2015-02-12 16:12:48,591 INFO  flow.FlowStep (BaseFlowStep.java:logInfo(834)) - [first] submitted hadoop job: job_local2080117784_0010
2015-02-12 16:12:48,591 INFO  mapred.LocalJobRunner (LocalJobRunner.java:createOutputCommitter(489)) - OutputCommitter is org.apache.hadoop.mapred.FileOutputCommitter
2015-02-12 16:12:48,591 INFO  flow.FlowStep (BaseFlowStep.java:logInfo(834)) - [first] tracking url: http://localhost:8080/
2015-02-12 16:12:48,593 INFO  mapred.LocalJobRunner (LocalJobRunner.java:runTasks(448)) - Waiting for map tasks
2015-02-12 16:12:48,593 INFO  mapred.LocalJobRunner (LocalJobRunner.java:run(224)) - Starting task: attempt_local2080117784_0010_m_000000_0
2015-02-12 16:12:48,594 INFO  util.ProcfsBasedProcessTree (ProcfsBasedProcessTree.java:isAvailable(181)) - ProcfsBasedProcessTree currently is supported only on Linux.
2015-02-12 16:12:48,594 INFO  mapred.Task (Task.java:initialize(587)) -  Using ResourceCalculatorProcessTree : null
2015-02-12 16:12:48,595 INFO  io.MultiInputSplit (MultiInputSplit.java:readFields(161)) - current split input path: file:/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/classes/test/data/lhs+rhs-cross.txt
2015-02-12 16:12:48,595 INFO  mapred.MapTask (MapTask.java:updateJobWithSplit(482)) - Processing split: cascading.tap.hadoop.io.MultiInputSplit@676e582a
2015-02-12 16:12:48,595 INFO  mapred.MapTask (MapTask.java:runOldMapper(436)) - numReduceTasks: 1
2015-02-12 16:12:48,611 INFO  mapred.MapTask (MapTask.java:setEquator(1202)) - (EQUATOR) 0 kvi 26214396(104857584)
2015-02-12 16:12:48,611 INFO  mapred.MapTask (MapTask.java:init(995)) - mapreduce.task.io.sort.mb: 100
2015-02-12 16:12:48,611 INFO  mapred.MapTask (MapTask.java:init(996)) - soft limit at 83886080
2015-02-12 16:12:48,611 INFO  mapred.MapTask (MapTask.java:init(997)) - bufstart = 0; bufvoid = 104857600
2015-02-12 16:12:48,611 INFO  mapred.MapTask (MapTask.java:init(998)) - kvstart = 26214396; length = 6553600
2015-02-12 16:12:48,613 INFO  mapred.MapTask (MapTask.java:createSortingCollector(402)) - Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer
2015-02-12 16:12:48,613 INFO  hadoop.FlowMapper (FlowMapper.java:configure(71)) - cascading version: 2.6.3
2015-02-12 16:12:48,613 INFO  hadoop.FlowMapper (FlowMapper.java:configure(72)) - child jvm opts: -Xmx200m
2015-02-12 16:12:48,618 INFO  hadoop.FlowMapper (FlowMapper.java:configure(87)) - sourcing from: Hfs["TextDelimited[['num', 'lower', 'upper']]"]["/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/classes/test/data/lhs+rhs-cross.txt"]
2015-02-12 16:12:48,618 INFO  hadoop.FlowMapper (FlowMapper.java:configure(90)) - sinking to: GroupBy(first)[by:[{1}:'num']]
2015-02-12 16:12:48,618 INFO  assembly.AggregateBy (AggregateBy.java:prepare(267)) - using threshold value: 2
2015-02-12 16:12:48,620 INFO  assembly.AggregateBy (AggregateBy.java:removeEldestEntry(327)) - flushed keys num times: 1, with threshold: 2
2015-02-12 16:12:48,620 INFO  assembly.AggregateBy (AggregateBy.java:removeEldestEntry(328)) - mem on flush (mb), free: 200, total: 509, max: 672
2015-02-12 16:12:48,620 INFO  assembly.AggregateBy (AggregateBy.java:removeEldestEntry(333)) - total mem is 75% of max mem, to better utilize unused memory consider increasing current LRU threshold with system property "cascading.aggregateby.threshold"
2015-02-12 16:12:48,621 INFO  assembly.AggregateBy (AggregateBy.java:removeEldestEntry(327)) - flushed keys num times: 3, with threshold: 2
2015-02-12 16:12:48,621 INFO  assembly.AggregateBy (AggregateBy.java:removeEldestEntry(328)) - mem on flush (mb), free: 199, total: 509, max: 672
2015-02-12 16:12:48,621 INFO  assembly.AggregateBy (AggregateBy.java:removeEldestEntry(333)) - total mem is 75% of max mem, to better utilize unused memory consider increasing current LRU threshold with system property "cascading.aggregateby.threshold"
2015-02-12 16:12:48,622 INFO  mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) - 
2015-02-12 16:12:48,623 INFO  mapred.MapTask (MapTask.java:flush(1457)) - Starting flush of map output
2015-02-12 16:12:48,623 INFO  mapred.MapTask (MapTask.java:flush(1475)) - Spilling map output
2015-02-12 16:12:48,623 INFO  mapred.MapTask (MapTask.java:flush(1476)) - bufstart = 0; bufend = 170; bufvoid = 104857600
2015-02-12 16:12:48,623 INFO  mapred.MapTask (MapTask.java:flush(1478)) - kvstart = 26214396(104857584); kvend = 26214380(104857520); length = 17/6553600
2015-02-12 16:12:48,624 INFO  mapred.MapTask (MapTask.java:sortAndSpill(1660)) - Finished spill 0
2015-02-12 16:12:48,626 INFO  mapred.Task (Task.java:done(1001)) - Task:attempt_local2080117784_0010_m_000000_0 is done. And is in the process of committing
2015-02-12 16:12:48,627 INFO  mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) - file:/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/classes/test/data/lhs+rhs-cross.txt:0+221
2015-02-12 16:12:48,627 INFO  mapred.Task (Task.java:sendDone(1121)) - Task 'attempt_local2080117784_0010_m_000000_0' done.
2015-02-12 16:12:48,627 INFO  mapred.LocalJobRunner (LocalJobRunner.java:run(249)) - Finishing task: attempt_local2080117784_0010_m_000000_0
2015-02-12 16:12:48,627 INFO  mapred.LocalJobRunner (LocalJobRunner.java:runTasks(456)) - map task executor complete.
2015-02-12 16:12:48,628 INFO  mapred.LocalJobRunner (LocalJobRunner.java:runTasks(448)) - Waiting for reduce tasks
2015-02-12 16:12:48,628 INFO  mapred.LocalJobRunner (LocalJobRunner.java:run(302)) - Starting task: attempt_local2080117784_0010_r_000000_0
2015-02-12 16:12:48,629 INFO  util.ProcfsBasedProcessTree (ProcfsBasedProcessTree.java:isAvailable(181)) - ProcfsBasedProcessTree currently is supported only on Linux.
2015-02-12 16:12:48,629 INFO  mapred.Task (Task.java:initialize(587)) -  Using ResourceCalculatorProcessTree : null
2015-02-12 16:12:48,629 INFO  mapred.ReduceTask (ReduceTask.java:run(362)) - Using ShuffleConsumerPlugin: org.apache.hadoop.mapreduce.task.reduce.Shuffle@21c88db
2015-02-12 16:12:48,630 INFO  reduce.MergeManagerImpl (MergeManagerImpl.java:<init>(195)) - MergerManager: memoryLimit=493250144, maxSingleShuffleLimit=123312536, mergeThreshold=325545120, ioSortFactor=10, memToMemMergeOutputsThreshold=10
2015-02-12 16:12:48,630 INFO  reduce.EventFetcher (EventFetcher.java:run(61)) - attempt_local2080117784_0010_r_000000_0 Thread started: EventFetcher for fetching Map Completion Events
2015-02-12 16:12:48,631 INFO  reduce.LocalFetcher (LocalFetcher.java:copyMapOutput(141)) - localfetcher#9 about to shuffle output of map attempt_local2080117784_0010_m_000000_0 decomp: 182 len: 186 to MEMORY
2015-02-12 16:12:48,631 INFO  reduce.InMemoryMapOutput (InMemoryMapOutput.java:shuffle(100)) - Read 182 bytes from map-output for attempt_local2080117784_0010_m_000000_0
2015-02-12 16:12:48,631 INFO  reduce.MergeManagerImpl (MergeManagerImpl.java:closeInMemoryFile(313)) - closeInMemoryFile -> map-output of size: 182, inMemoryMapOutputs.size() -> 1, commitMemory -> 0, usedMemory ->182
2015-02-12 16:12:48,632 INFO  reduce.EventFetcher (EventFetcher.java:run(76)) - EventFetcher is interrupted.. Returning
2015-02-12 16:12:48,632 INFO  mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) - 1 / 1 copied.
2015-02-12 16:12:48,633 INFO  reduce.MergeManagerImpl (MergeManagerImpl.java:finalMerge(673)) - finalMerge called with 1 in-memory map-outputs and 0 on-disk map-outputs
2015-02-12 16:12:48,634 INFO  mapred.Merger (Merger.java:merge(606)) - Merging 1 sorted segments
2015-02-12 16:12:48,634 INFO  mapred.Merger (Merger.java:merge(705)) - Down to the last merge-pass, with 1 segments left of total size: 160 bytes
2015-02-12 16:12:48,636 INFO  reduce.MergeManagerImpl (MergeManagerImpl.java:finalMerge(750)) - Merged 1 segments, 182 bytes to disk to satisfy reduce memory limit
2015-02-12 16:12:48,636 INFO  reduce.MergeManagerImpl (MergeManagerImpl.java:finalMerge(780)) - Merging 1 files, 186 bytes from disk
2015-02-12 16:12:48,636 INFO  reduce.MergeManagerImpl (MergeManagerImpl.java:finalMerge(795)) - Merging 0 segments, 0 bytes from memory into reduce
2015-02-12 16:12:48,636 INFO  mapred.Merger (Merger.java:merge(606)) - Merging 1 sorted segments
2015-02-12 16:12:48,636 INFO  mapred.Merger (Merger.java:merge(705)) - Down to the last merge-pass, with 1 segments left of total size: 160 bytes
2015-02-12 16:12:48,637 INFO  mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) - 1 / 1 copied.
2015-02-12 16:12:48,637 INFO  hadoop.FlowReducer (FlowReducer.java:configure(78)) - cascading version: 2.6.3
2015-02-12 16:12:48,637 INFO  hadoop.FlowReducer (FlowReducer.java:configure(79)) - child jvm opts: -Xmx200m
2015-02-12 16:12:48,643 INFO  hadoop.FlowReducer (FlowReducer.java:configure(97)) - sourcing from: GroupBy(first)[by:[{1}:'num']]
2015-02-12 16:12:48,643 INFO  hadoop.FlowReducer (FlowReducer.java:configure(100)) - sinking to: Hfs["TextDelimited[['num', 'lower', 'upper']]"]["/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/test/output/hadoop2-mr1/assemblyhelpersplatform/firstnfields"]
2015-02-12 16:12:48,651 INFO  mapred.Task (Task.java:done(1001)) - Task:attempt_local2080117784_0010_r_000000_0 is done. And is in the process of committing
2015-02-12 16:12:48,652 INFO  mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) - 1 / 1 copied.
2015-02-12 16:12:48,652 INFO  mapred.Task (Task.java:commit(1162)) - Task attempt_local2080117784_0010_r_000000_0 is allowed to commit now
2015-02-12 16:12:48,653 INFO  output.FileOutputCommitter (FileOutputCommitter.java:commitTask(439)) - Saved output of task 'attempt_local2080117784_0010_r_000000_0' to file:/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/test/output/hadoop2-mr1/assemblyhelpersplatform/firstnfields/_temporary/0/task_local2080117784_0010_r_000000
2015-02-12 16:12:48,653 INFO  mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) - reduce > reduce
2015-02-12 16:12:48,653 INFO  mapred.Task (Task.java:sendDone(1121)) - Task 'attempt_local2080117784_0010_r_000000_0' done.
2015-02-12 16:12:48,654 INFO  mapred.LocalJobRunner (LocalJobRunner.java:run(325)) - Finishing task: attempt_local2080117784_0010_r_000000_0
2015-02-12 16:12:48,654 INFO  mapred.LocalJobRunner (LocalJobRunner.java:runTasks(456)) - reduce task executor complete.
2015-02-12 16:12:48,673 INFO  util.Hadoop18TapUtil (Hadoop18TapUtil.java:cleanTempPath(219)) - deleting temp path /Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/test/output/hadoop2-mr1/assemblyhelpersplatform/firstnfields/_temporary
2015-02-12 16:12:48,695 INFO  mapred.FileInputFormat (FileInputFormat.java:listStatus(247)) - Total input paths to process : 1
2015-02-12 16:12:48,696 INFO  cascading.PlatformTestCase (PlatformTestCase.java:tearDown(209)) - copying to local /Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/test/output//hadoop2-mr1/assemblyhelpersplatform/firstnfields
2015-02-12 16:12:48,711 INFO  util.HadoopUtil (HadoopUtil.java:findMainClass(336)) - using default application jar, may cause class not found exceptions on the cluster
2015-02-12 16:12:48,711 INFO  planner.HadoopPlanner (HadoopPlanner.java:initialize(225)) - using application jar: /Users/soberoi/.gradle/caches/modules-2/files-2.1/cascading/cascading-hadoop2-mr1/2.6.3/9ae016e2b8d246b9c1de9168e715d9f1b971aa4a/cascading-hadoop2-mr1-2.6.3.jar
2015-02-12 16:12:48,716 INFO  Configuration.deprecation (Configuration.java:warnOnceIfDeprecated(1022)) - mapred.used.genericoptionsparser is deprecated. Instead, use mapreduce.client.genericoptionsparser.used
2015-02-12 16:12:48,717 INFO  Configuration.deprecation (Configuration.java:warnOnceIfDeprecated(1022)) - mapred.job.tracker is deprecated. Instead, use mapreduce.jobtracker.address
2015-02-12 16:12:48,717 INFO  Configuration.deprecation (Configuration.java:warnOnceIfDeprecated(1022)) - mapred.output.compress is deprecated. Instead, use mapreduce.output.fileoutputformat.compress
2015-02-12 16:12:48,723 INFO  flow.Flow (BaseFlow.java:logInfo(1378)) - [coerce] starting
2015-02-12 16:12:48,724 INFO  flow.Flow (BaseFlow.java:logInfo(1378)) - [coerce]  source: Hfs["TextLine[['offset', 'line']->[ALL]]"]["/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/classes/test/data/lower.txt"]
2015-02-12 16:12:48,724 INFO  flow.Flow (BaseFlow.java:logInfo(1378)) - [coerce]  sink: Hfs["TextLine[['line']->['num', 'char']]"]["/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/test/output/hadoop2-mr1/assemblyhelpersplatform/coercefields"]
2015-02-12 16:12:48,724 INFO  flow.Flow (BaseFlow.java:logInfo(1378)) - [coerce]  parallel execution is enabled: false
2015-02-12 16:12:48,724 INFO  flow.Flow (BaseFlow.java:logInfo(1378)) - [coerce]  starting jobs: 1
2015-02-12 16:12:48,724 INFO  flow.Flow (BaseFlow.java:logInfo(1378)) - [coerce]  allocating threads: 1
2015-02-12 16:12:48,724 INFO  flow.FlowStep (BaseFlowStep.java:logInfo(834)) - [coerce] starting step: (1/1) ...persplatform/coercefields
2015-02-12 16:12:48,725 INFO  jvm.JvmMetrics (JvmMetrics.java:init(71)) - Cannot initialize JVM Metrics with processName=JobTracker, sessionId= - already initialized
2015-02-12 16:12:48,725 INFO  jvm.JvmMetrics (JvmMetrics.java:init(71)) - Cannot initialize JVM Metrics with processName=JobTracker, sessionId= - already initialized
2015-02-12 16:12:48,743 INFO  mapred.FileInputFormat (FileInputFormat.java:listStatus(247)) - Total input paths to process : 1
2015-02-12 16:12:48,758 INFO  mapreduce.JobSubmitter (JobSubmitter.java:submitJobInternal(492)) - number of splits:1
2015-02-12 16:12:48,770 INFO  mapreduce.JobSubmitter (JobSubmitter.java:printTokens(575)) - Submitting tokens for job: job_local994598886_0011
2015-02-12 16:12:48,780 WARN  conf.Configuration (Configuration.java:loadProperty(2510)) - file:/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/tmp/cascading/staging/soberoi994598886/.staging/job_local994598886_0011/job.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.retry.interval;  Ignoring.
2015-02-12 16:12:48,781 WARN  conf.Configuration (Configuration.java:loadProperty(2510)) - file:/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/tmp/cascading/staging/soberoi994598886/.staging/job_local994598886_0011/job.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.attempts;  Ignoring.
2015-02-12 16:12:48,812 WARN  conf.Configuration (Configuration.java:loadProperty(2510)) - file:/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/test/tmp/mapred/local/localRunner/soberoi/job_local994598886_0011/job_local994598886_0011.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.retry.interval;  Ignoring.
2015-02-12 16:12:48,813 WARN  conf.Configuration (Configuration.java:loadProperty(2510)) - file:/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/test/tmp/mapred/local/localRunner/soberoi/job_local994598886_0011/job_local994598886_0011.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.attempts;  Ignoring.
2015-02-12 16:12:48,813 INFO  mapreduce.Job (Job.java:submit(1299)) - The url to track the job: http://localhost:8080/
2015-02-12 16:12:48,813 INFO  mapred.LocalJobRunner (LocalJobRunner.java:createOutputCommitter(471)) - OutputCommitter set in config null
2015-02-12 16:12:48,814 INFO  flow.FlowStep (BaseFlowStep.java:logInfo(834)) - [coerce] submitted hadoop job: job_local994598886_0011
2015-02-12 16:12:48,814 INFO  mapred.LocalJobRunner (LocalJobRunner.java:createOutputCommitter(489)) - OutputCommitter is org.apache.hadoop.mapred.FileOutputCommitter
2015-02-12 16:12:48,814 INFO  flow.FlowStep (BaseFlowStep.java:logInfo(834)) - [coerce] tracking url: http://localhost:8080/
2015-02-12 16:12:48,815 INFO  mapred.LocalJobRunner (LocalJobRunner.java:runTasks(448)) - Waiting for map tasks
2015-02-12 16:12:48,815 INFO  mapred.LocalJobRunner (LocalJobRunner.java:run(224)) - Starting task: attempt_local994598886_0011_m_000000_0
2015-02-12 16:12:48,816 INFO  util.ProcfsBasedProcessTree (ProcfsBasedProcessTree.java:isAvailable(181)) - ProcfsBasedProcessTree currently is supported only on Linux.
2015-02-12 16:12:48,816 INFO  mapred.Task (Task.java:initialize(587)) -  Using ResourceCalculatorProcessTree : null
2015-02-12 16:12:48,817 INFO  io.MultiInputSplit (MultiInputSplit.java:readFields(161)) - current split input path: file:/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/classes/test/data/lower.txt
2015-02-12 16:12:48,817 INFO  mapred.MapTask (MapTask.java:updateJobWithSplit(482)) - Processing split: cascading.tap.hadoop.io.MultiInputSplit@61a3e88e
2015-02-12 16:12:48,818 INFO  mapred.MapTask (MapTask.java:runOldMapper(436)) - numReduceTasks: 0
2015-02-12 16:12:48,823 INFO  hadoop.FlowMapper (FlowMapper.java:configure(71)) - cascading version: 2.6.3
2015-02-12 16:12:48,823 INFO  hadoop.FlowMapper (FlowMapper.java:configure(72)) - child jvm opts: -Xmx200m
2015-02-12 16:12:48,827 INFO  hadoop.FlowMapper (FlowMapper.java:configure(87)) - sourcing from: Hfs["TextLine[['offset', 'line']->[ALL]]"]["/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/classes/test/data/lower.txt"]
2015-02-12 16:12:48,828 INFO  hadoop.FlowMapper (FlowMapper.java:configure(90)) - sinking to: Hfs["TextLine[['line']->['num', 'char']]"]["/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/test/output/hadoop2-mr1/assemblyhelpersplatform/coercefields"]
2015-02-12 16:12:48,833 INFO  mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) - 
2015-02-12 16:12:48,834 INFO  mapred.Task (Task.java:done(1001)) - Task:attempt_local994598886_0011_m_000000_0 is done. And is in the process of committing
2015-02-12 16:12:48,835 INFO  mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) - 
2015-02-12 16:12:48,835 INFO  mapred.Task (Task.java:commit(1162)) - Task attempt_local994598886_0011_m_000000_0 is allowed to commit now
2015-02-12 16:12:48,835 INFO  output.FileOutputCommitter (FileOutputCommitter.java:commitTask(439)) - Saved output of task 'attempt_local994598886_0011_m_000000_0' to file:/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/test/output/hadoop2-mr1/assemblyhelpersplatform/coercefields/_temporary/0/task_local994598886_0011_m_000000
2015-02-12 16:12:48,836 INFO  mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) - file:/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/classes/test/data/lower.txt:0+19
2015-02-12 16:12:48,836 INFO  mapred.Task (Task.java:sendDone(1121)) - Task 'attempt_local994598886_0011_m_000000_0' done.
2015-02-12 16:12:48,836 INFO  mapred.LocalJobRunner (LocalJobRunner.java:run(249)) - Finishing task: attempt_local994598886_0011_m_000000_0
2015-02-12 16:12:48,836 INFO  mapred.LocalJobRunner (LocalJobRunner.java:runTasks(456)) - map task executor complete.
2015-02-12 16:12:48,854 INFO  util.Hadoop18TapUtil (Hadoop18TapUtil.java:cleanTempPath(219)) - deleting temp path /Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/test/output/hadoop2-mr1/assemblyhelpersplatform/coercefields/_temporary
2015-02-12 16:12:48,873 INFO  mapred.FileInputFormat (FileInputFormat.java:listStatus(247)) - Total input paths to process : 1
2015-02-12 16:12:48,874 INFO  cascading.PlatformTestCase (PlatformTestCase.java:tearDown(209)) - copying to local /Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/test/output//hadoop2-mr1/assemblyhelpersplatform/coercefields
2015-02-12 16:12:48,887 INFO  util.HadoopUtil (HadoopUtil.java:findMainClass(336)) - using default application jar, may cause class not found exceptions on the cluster
2015-02-12 16:12:48,887 INFO  planner.HadoopPlanner (HadoopPlanner.java:initialize(225)) - using application jar: /Users/soberoi/.gradle/caches/modules-2/files-2.1/cascading/cascading-hadoop2-mr1/2.6.3/9ae016e2b8d246b9c1de9168e715d9f1b971aa4a/cascading-hadoop2-mr1-2.6.3.jar
2015-02-12 16:12:48,892 INFO  Configuration.deprecation (Configuration.java:warnOnceIfDeprecated(1022)) - mapred.used.genericoptionsparser is deprecated. Instead, use mapreduce.client.genericoptionsparser.used
2015-02-12 16:12:48,894 INFO  Configuration.deprecation (Configuration.java:warnOnceIfDeprecated(1022)) - mapred.job.tracker is deprecated. Instead, use mapreduce.jobtracker.address
2015-02-12 16:12:48,895 INFO  Configuration.deprecation (Configuration.java:warnOnceIfDeprecated(1022)) - mapred.output.compress is deprecated. Instead, use mapreduce.output.fileoutputformat.compress
2015-02-12 16:12:48,900 INFO  flow.Flow (BaseFlow.java:logInfo(1378)) - [count] starting
2015-02-12 16:12:48,900 INFO  flow.Flow (BaseFlow.java:logInfo(1378)) - [count]  source: Hfs["TextDelimited[['num', 'char']]"]["/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/classes/test/data/lhs.txt"]
2015-02-12 16:12:48,901 INFO  flow.Flow (BaseFlow.java:logInfo(1378)) - [count]  sink: Hfs["TextDelimited[['char', 'count']]"]["/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/test/output/hadoop2-mr1/assemblyhelpersplatform/count"]
2015-02-12 16:12:48,901 INFO  flow.Flow (BaseFlow.java:logInfo(1378)) - [count]  parallel execution is enabled: false
2015-02-12 16:12:48,901 INFO  flow.Flow (BaseFlow.java:logInfo(1378)) - [count]  starting jobs: 1
2015-02-12 16:12:48,901 INFO  flow.Flow (BaseFlow.java:logInfo(1378)) - [count]  allocating threads: 1
2015-02-12 16:12:48,901 INFO  flow.FlowStep (BaseFlowStep.java:logInfo(834)) - [count] starting step: (1/1) ...mblyhelpersplatform/count
2015-02-12 16:12:48,902 INFO  jvm.JvmMetrics (JvmMetrics.java:init(71)) - Cannot initialize JVM Metrics with processName=JobTracker, sessionId= - already initialized
2015-02-12 16:12:48,902 INFO  jvm.JvmMetrics (JvmMetrics.java:init(71)) - Cannot initialize JVM Metrics with processName=JobTracker, sessionId= - already initialized
2015-02-12 16:12:48,921 INFO  mapred.FileInputFormat (FileInputFormat.java:listStatus(247)) - Total input paths to process : 1
2015-02-12 16:12:48,937 INFO  mapreduce.JobSubmitter (JobSubmitter.java:submitJobInternal(492)) - number of splits:1
2015-02-12 16:12:48,948 INFO  mapreduce.JobSubmitter (JobSubmitter.java:printTokens(575)) - Submitting tokens for job: job_local910453495_0012
2015-02-12 16:12:48,958 WARN  conf.Configuration (Configuration.java:loadProperty(2510)) - file:/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/tmp/cascading/staging/soberoi910453495/.staging/job_local910453495_0012/job.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.retry.interval;  Ignoring.
2015-02-12 16:12:48,959 WARN  conf.Configuration (Configuration.java:loadProperty(2510)) - file:/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/tmp/cascading/staging/soberoi910453495/.staging/job_local910453495_0012/job.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.attempts;  Ignoring.
2015-02-12 16:12:48,989 WARN  conf.Configuration (Configuration.java:loadProperty(2510)) - file:/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/test/tmp/mapred/local/localRunner/soberoi/job_local910453495_0012/job_local910453495_0012.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.retry.interval;  Ignoring.
2015-02-12 16:12:48,990 WARN  conf.Configuration (Configuration.java:loadProperty(2510)) - file:/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/test/tmp/mapred/local/localRunner/soberoi/job_local910453495_0012/job_local910453495_0012.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.attempts;  Ignoring.
2015-02-12 16:12:48,990 INFO  mapreduce.Job (Job.java:submit(1299)) - The url to track the job: http://localhost:8080/
2015-02-12 16:12:48,991 INFO  mapred.LocalJobRunner (LocalJobRunner.java:createOutputCommitter(471)) - OutputCommitter set in config null
2015-02-12 16:12:48,991 INFO  mapred.LocalJobRunner (LocalJobRunner.java:createOutputCommitter(489)) - OutputCommitter is org.apache.hadoop.mapred.FileOutputCommitter
2015-02-12 16:12:48,991 INFO  flow.FlowStep (BaseFlowStep.java:logInfo(834)) - [count] submitted hadoop job: job_local910453495_0012
2015-02-12 16:12:48,991 INFO  flow.FlowStep (BaseFlowStep.java:logInfo(834)) - [count] tracking url: http://localhost:8080/
2015-02-12 16:12:48,993 INFO  mapred.LocalJobRunner (LocalJobRunner.java:runTasks(448)) - Waiting for map tasks
2015-02-12 16:12:48,993 INFO  mapred.LocalJobRunner (LocalJobRunner.java:run(224)) - Starting task: attempt_local910453495_0012_m_000000_0
2015-02-12 16:12:48,994 INFO  util.ProcfsBasedProcessTree (ProcfsBasedProcessTree.java:isAvailable(181)) - ProcfsBasedProcessTree currently is supported only on Linux.
2015-02-12 16:12:48,994 INFO  mapred.Task (Task.java:initialize(587)) -  Using ResourceCalculatorProcessTree : null
2015-02-12 16:12:48,994 INFO  io.MultiInputSplit (MultiInputSplit.java:readFields(161)) - current split input path: file:/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/classes/test/data/lhs.txt
2015-02-12 16:12:48,994 INFO  mapred.MapTask (MapTask.java:updateJobWithSplit(482)) - Processing split: cascading.tap.hadoop.io.MultiInputSplit@4db1f211
2015-02-12 16:12:48,995 INFO  mapred.MapTask (MapTask.java:runOldMapper(436)) - numReduceTasks: 1
2015-02-12 16:12:49,012 INFO  mapred.MapTask (MapTask.java:setEquator(1202)) - (EQUATOR) 0 kvi 26214396(104857584)
2015-02-12 16:12:49,012 INFO  mapred.MapTask (MapTask.java:init(995)) - mapreduce.task.io.sort.mb: 100
2015-02-12 16:12:49,012 INFO  mapred.MapTask (MapTask.java:init(996)) - soft limit at 83886080
2015-02-12 16:12:49,012 INFO  mapred.MapTask (MapTask.java:init(997)) - bufstart = 0; bufvoid = 104857600
2015-02-12 16:12:49,013 INFO  mapred.MapTask (MapTask.java:init(998)) - kvstart = 26214396; length = 6553600
2015-02-12 16:12:49,014 INFO  mapred.MapTask (MapTask.java:createSortingCollector(402)) - Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer
2015-02-12 16:12:49,014 INFO  hadoop.FlowMapper (FlowMapper.java:configure(71)) - cascading version: 2.6.3
2015-02-12 16:12:49,014 INFO  hadoop.FlowMapper (FlowMapper.java:configure(72)) - child jvm opts: -Xmx200m
2015-02-12 16:12:49,019 INFO  hadoop.FlowMapper (FlowMapper.java:configure(87)) - sourcing from: Hfs["TextDelimited[['num', 'char']]"]["/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/classes/test/data/lhs.txt"]
2015-02-12 16:12:49,019 INFO  hadoop.FlowMapper (FlowMapper.java:configure(90)) - sinking to: GroupBy(count)[by:[{1}:'char']]
2015-02-12 16:12:49,019 INFO  assembly.AggregateBy (AggregateBy.java:prepare(267)) - using threshold value: 2
2015-02-12 16:12:49,020 INFO  assembly.AggregateBy (AggregateBy.java:removeEldestEntry(327)) - flushed keys num times: 1, with threshold: 2
2015-02-12 16:12:49,020 INFO  assembly.AggregateBy (AggregateBy.java:removeEldestEntry(328)) - mem on flush (mb), free: 199, total: 509, max: 672
2015-02-12 16:12:49,020 INFO  assembly.AggregateBy (AggregateBy.java:removeEldestEntry(333)) - total mem is 75% of max mem, to better utilize unused memory consider increasing current LRU threshold with system property "cascading.aggregateby.threshold"
2015-02-12 16:12:49,021 INFO  assembly.AggregateBy (AggregateBy.java:removeEldestEntry(327)) - flushed keys num times: 3, with threshold: 2
2015-02-12 16:12:49,021 INFO  assembly.AggregateBy (AggregateBy.java:removeEldestEntry(328)) - mem on flush (mb), free: 199, total: 509, max: 672
2015-02-12 16:12:49,021 INFO  assembly.AggregateBy (AggregateBy.java:removeEldestEntry(333)) - total mem is 75% of max mem, to better utilize unused memory consider increasing current LRU threshold with system property "cascading.aggregateby.threshold"
2015-02-12 16:12:49,021 INFO  assembly.AggregateBy (AggregateBy.java:removeEldestEntry(327)) - flushed keys num times: 5, with threshold: 2
2015-02-12 16:12:49,021 INFO  assembly.AggregateBy (AggregateBy.java:removeEldestEntry(328)) - mem on flush (mb), free: 199, total: 509, max: 672
2015-02-12 16:12:49,021 INFO  assembly.AggregateBy (AggregateBy.java:removeEldestEntry(333)) - total mem is 75% of max mem, to better utilize unused memory consider increasing current LRU threshold with system property "cascading.aggregateby.threshold"
2015-02-12 16:12:49,021 INFO  assembly.AggregateBy (AggregateBy.java:removeEldestEntry(327)) - flushed keys num times: 7, with threshold: 2
2015-02-12 16:12:49,022 INFO  assembly.AggregateBy (AggregateBy.java:removeEldestEntry(328)) - mem on flush (mb), free: 199, total: 509, max: 672
2015-02-12 16:12:49,022 INFO  assembly.AggregateBy (AggregateBy.java:removeEldestEntry(333)) - total mem is 75% of max mem, to better utilize unused memory consider increasing current LRU threshold with system property "cascading.aggregateby.threshold"
2015-02-12 16:12:49,022 INFO  mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) - 
2015-02-12 16:12:49,023 INFO  mapred.MapTask (MapTask.java:flush(1457)) - Starting flush of map output
2015-02-12 16:12:49,023 INFO  mapred.MapTask (MapTask.java:flush(1475)) - Spilling map output
2015-02-12 16:12:49,023 INFO  mapred.MapTask (MapTask.java:flush(1476)) - bufstart = 0; bufend = 99; bufvoid = 104857600
2015-02-12 16:12:49,023 INFO  mapred.MapTask (MapTask.java:flush(1478)) - kvstart = 26214396(104857584); kvend = 26214364(104857456); length = 33/6553600
2015-02-12 16:12:49,024 INFO  mapred.MapTask (MapTask.java:sortAndSpill(1660)) - Finished spill 0
2015-02-12 16:12:49,025 INFO  mapred.Task (Task.java:done(1001)) - Task:attempt_local910453495_0012_m_000000_0 is done. And is in the process of committing
2015-02-12 16:12:49,027 INFO  mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) - file:/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/classes/test/data/lhs.txt:0+52
2015-02-12 16:12:49,027 INFO  mapred.Task (Task.java:sendDone(1121)) - Task 'attempt_local910453495_0012_m_000000_0' done.
2015-02-12 16:12:49,027 INFO  mapred.LocalJobRunner (LocalJobRunner.java:run(249)) - Finishing task: attempt_local910453495_0012_m_000000_0
2015-02-12 16:12:49,027 INFO  mapred.LocalJobRunner (LocalJobRunner.java:runTasks(456)) - map task executor complete.
2015-02-12 16:12:49,027 INFO  mapred.LocalJobRunner (LocalJobRunner.java:runTasks(448)) - Waiting for reduce tasks
2015-02-12 16:12:49,027 INFO  mapred.LocalJobRunner (LocalJobRunner.java:run(302)) - Starting task: attempt_local910453495_0012_r_000000_0
2015-02-12 16:12:49,028 INFO  util.ProcfsBasedProcessTree (ProcfsBasedProcessTree.java:isAvailable(181)) - ProcfsBasedProcessTree currently is supported only on Linux.
2015-02-12 16:12:49,029 INFO  mapred.Task (Task.java:initialize(587)) -  Using ResourceCalculatorProcessTree : null
2015-02-12 16:12:49,029 INFO  mapred.ReduceTask (ReduceTask.java:run(362)) - Using ShuffleConsumerPlugin: org.apache.hadoop.mapreduce.task.reduce.Shuffle@202a6eeb
2015-02-12 16:12:49,029 INFO  reduce.MergeManagerImpl (MergeManagerImpl.java:<init>(195)) - MergerManager: memoryLimit=493250144, maxSingleShuffleLimit=123312536, mergeThreshold=325545120, ioSortFactor=10, memToMemMergeOutputsThreshold=10
2015-02-12 16:12:49,030 INFO  reduce.EventFetcher (EventFetcher.java:run(61)) - attempt_local910453495_0012_r_000000_0 Thread started: EventFetcher for fetching Map Completion Events
2015-02-12 16:12:49,030 INFO  reduce.LocalFetcher (LocalFetcher.java:copyMapOutput(141)) - localfetcher#10 about to shuffle output of map attempt_local910453495_0012_m_000000_0 decomp: 119 len: 123 to MEMORY
2015-02-12 16:12:49,031 INFO  reduce.InMemoryMapOutput (InMemoryMapOutput.java:shuffle(100)) - Read 119 bytes from map-output for attempt_local910453495_0012_m_000000_0
2015-02-12 16:12:49,031 INFO  reduce.MergeManagerImpl (MergeManagerImpl.java:closeInMemoryFile(313)) - closeInMemoryFile -> map-output of size: 119, inMemoryMapOutputs.size() -> 1, commitMemory -> 0, usedMemory ->119
2015-02-12 16:12:49,031 INFO  reduce.EventFetcher (EventFetcher.java:run(76)) - EventFetcher is interrupted.. Returning
2015-02-12 16:12:49,032 INFO  mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) - 1 / 1 copied.
2015-02-12 16:12:49,032 INFO  reduce.MergeManagerImpl (MergeManagerImpl.java:finalMerge(673)) - finalMerge called with 1 in-memory map-outputs and 0 on-disk map-outputs
2015-02-12 16:12:49,033 INFO  mapred.Merger (Merger.java:merge(606)) - Merging 1 sorted segments
2015-02-12 16:12:49,033 INFO  mapred.Merger (Merger.java:merge(705)) - Down to the last merge-pass, with 1 segments left of total size: 110 bytes
2015-02-12 16:12:49,034 INFO  reduce.MergeManagerImpl (MergeManagerImpl.java:finalMerge(750)) - Merged 1 segments, 119 bytes to disk to satisfy reduce memory limit
2015-02-12 16:12:49,034 INFO  reduce.MergeManagerImpl (MergeManagerImpl.java:finalMerge(780)) - Merging 1 files, 123 bytes from disk
2015-02-12 16:12:49,034 INFO  reduce.MergeManagerImpl (MergeManagerImpl.java:finalMerge(795)) - Merging 0 segments, 0 bytes from memory into reduce
2015-02-12 16:12:49,034 INFO  mapred.Merger (Merger.java:merge(606)) - Merging 1 sorted segments
2015-02-12 16:12:49,035 INFO  mapred.Merger (Merger.java:merge(705)) - Down to the last merge-pass, with 1 segments left of total size: 110 bytes
2015-02-12 16:12:49,035 INFO  mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) - 1 / 1 copied.
2015-02-12 16:12:49,035 INFO  hadoop.FlowReducer (FlowReducer.java:configure(78)) - cascading version: 2.6.3
2015-02-12 16:12:49,036 INFO  hadoop.FlowReducer (FlowReducer.java:configure(79)) - child jvm opts: -Xmx200m
2015-02-12 16:12:49,042 INFO  hadoop.FlowReducer (FlowReducer.java:configure(97)) - sourcing from: GroupBy(count)[by:[{1}:'char']]
2015-02-12 16:12:49,042 INFO  hadoop.FlowReducer (FlowReducer.java:configure(100)) - sinking to: Hfs["TextDelimited[['char', 'count']]"]["/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/test/output/hadoop2-mr1/assemblyhelpersplatform/count"]
2015-02-12 16:12:49,049 INFO  mapred.Task (Task.java:done(1001)) - Task:attempt_local910453495_0012_r_000000_0 is done. And is in the process of committing
2015-02-12 16:12:49,050 INFO  mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) - 1 / 1 copied.
2015-02-12 16:12:49,050 INFO  mapred.Task (Task.java:commit(1162)) - Task attempt_local910453495_0012_r_000000_0 is allowed to commit now
2015-02-12 16:12:49,051 INFO  output.FileOutputCommitter (FileOutputCommitter.java:commitTask(439)) - Saved output of task 'attempt_local910453495_0012_r_000000_0' to file:/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/test/output/hadoop2-mr1/assemblyhelpersplatform/count/_temporary/0/task_local910453495_0012_r_000000
2015-02-12 16:12:49,052 INFO  mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) - reduce > reduce
2015-02-12 16:12:49,052 INFO  mapred.Task (Task.java:sendDone(1121)) - Task 'attempt_local910453495_0012_r_000000_0' done.
2015-02-12 16:12:49,052 INFO  mapred.LocalJobRunner (LocalJobRunner.java:run(325)) - Finishing task: attempt_local910453495_0012_r_000000_0
2015-02-12 16:12:49,052 INFO  mapred.LocalJobRunner (LocalJobRunner.java:runTasks(456)) - reduce task executor complete.
2015-02-12 16:12:49,070 INFO  util.Hadoop18TapUtil (Hadoop18TapUtil.java:cleanTempPath(219)) - deleting temp path /Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/test/output/hadoop2-mr1/assemblyhelpersplatform/count/_temporary
2015-02-12 16:12:49,091 INFO  mapred.FileInputFormat (FileInputFormat.java:listStatus(247)) - Total input paths to process : 1
2015-02-12 16:12:49,113 INFO  mapred.FileInputFormat (FileInputFormat.java:listStatus(247)) - Total input paths to process : 1
2015-02-12 16:12:49,114 INFO  cascading.PlatformTestCase (PlatformTestCase.java:tearDown(209)) - copying to local /Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/test/output//hadoop2-mr1/assemblyhelpersplatform/count
2015-02-12 16:12:49,132 INFO  util.HadoopUtil (HadoopUtil.java:findMainClass(336)) - using default application jar, may cause class not found exceptions on the cluster
2015-02-12 16:12:49,132 INFO  planner.HadoopPlanner (HadoopPlanner.java:initialize(225)) - using application jar: /Users/soberoi/.gradle/caches/modules-2/files-2.1/cascading/cascading-hadoop2-mr1/2.6.3/9ae016e2b8d246b9c1de9168e715d9f1b971aa4a/cascading-hadoop2-mr1-2.6.3.jar
2015-02-12 16:12:49,137 INFO  Configuration.deprecation (Configuration.java:warnOnceIfDeprecated(1022)) - mapred.used.genericoptionsparser is deprecated. Instead, use mapreduce.client.genericoptionsparser.used
2015-02-12 16:12:49,138 INFO  Configuration.deprecation (Configuration.java:warnOnceIfDeprecated(1022)) - mapred.job.tracker is deprecated. Instead, use mapreduce.jobtracker.address
2015-02-12 16:12:49,138 INFO  Configuration.deprecation (Configuration.java:warnOnceIfDeprecated(1022)) - mapred.output.compress is deprecated. Instead, use mapreduce.output.fileoutputformat.compress
2015-02-12 16:12:49,143 INFO  flow.Flow (BaseFlow.java:logInfo(1378)) - [max] starting
2015-02-12 16:12:49,143 INFO  flow.Flow (BaseFlow.java:logInfo(1378)) - [max]  source: Hfs["TextDelimited[['num', 'char']]"]["/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/classes/test/data/lhs.txt"]
2015-02-12 16:12:49,143 INFO  flow.Flow (BaseFlow.java:logInfo(1378)) - [max]  sink: Hfs["TextDelimited[['char', 'max']]"]["/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/test/output/hadoop2-mr1/assemblyhelpersplatform/maxby"]
2015-02-12 16:12:49,143 INFO  flow.Flow (BaseFlow.java:logInfo(1378)) - [max]  parallel execution is enabled: false
2015-02-12 16:12:49,144 INFO  flow.Flow (BaseFlow.java:logInfo(1378)) - [max]  starting jobs: 1
2015-02-12 16:12:49,144 INFO  flow.Flow (BaseFlow.java:logInfo(1378)) - [max]  allocating threads: 1
2015-02-12 16:12:49,144 INFO  flow.FlowStep (BaseFlowStep.java:logInfo(834)) - [max] starting step: (1/1) ...mblyhelpersplatform/maxby
2015-02-12 16:12:49,145 INFO  jvm.JvmMetrics (JvmMetrics.java:init(71)) - Cannot initialize JVM Metrics with processName=JobTracker, sessionId= - already initialized
2015-02-12 16:12:49,146 INFO  jvm.JvmMetrics (JvmMetrics.java:init(71)) - Cannot initialize JVM Metrics with processName=JobTracker, sessionId= - already initialized
2015-02-12 16:12:49,167 INFO  mapred.FileInputFormat (FileInputFormat.java:listStatus(247)) - Total input paths to process : 1
2015-02-12 16:12:49,185 INFO  mapreduce.JobSubmitter (JobSubmitter.java:submitJobInternal(492)) - number of splits:1
2015-02-12 16:12:49,197 INFO  mapreduce.JobSubmitter (JobSubmitter.java:printTokens(575)) - Submitting tokens for job: job_local1579825300_0013
2015-02-12 16:12:49,213 WARN  conf.Configuration (Configuration.java:loadProperty(2510)) - file:/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/tmp/cascading/staging/soberoi1579825300/.staging/job_local1579825300_0013/job.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.retry.interval;  Ignoring.
2015-02-12 16:12:49,214 WARN  conf.Configuration (Configuration.java:loadProperty(2510)) - file:/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/tmp/cascading/staging/soberoi1579825300/.staging/job_local1579825300_0013/job.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.attempts;  Ignoring.
2015-02-12 16:12:49,247 WARN  conf.Configuration (Configuration.java:loadProperty(2510)) - file:/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/test/tmp/mapred/local/localRunner/soberoi/job_local1579825300_0013/job_local1579825300_0013.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.retry.interval;  Ignoring.
2015-02-12 16:12:49,248 WARN  conf.Configuration (Configuration.java:loadProperty(2510)) - file:/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/test/tmp/mapred/local/localRunner/soberoi/job_local1579825300_0013/job_local1579825300_0013.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.attempts;  Ignoring.
2015-02-12 16:12:49,248 INFO  mapreduce.Job (Job.java:submit(1299)) - The url to track the job: http://localhost:8080/
2015-02-12 16:12:49,249 INFO  mapred.LocalJobRunner (LocalJobRunner.java:createOutputCommitter(471)) - OutputCommitter set in config null
2015-02-12 16:12:49,249 INFO  flow.FlowStep (BaseFlowStep.java:logInfo(834)) - [max] submitted hadoop job: job_local1579825300_0013
2015-02-12 16:12:49,249 INFO  mapred.LocalJobRunner (LocalJobRunner.java:createOutputCommitter(489)) - OutputCommitter is org.apache.hadoop.mapred.FileOutputCommitter
2015-02-12 16:12:49,249 INFO  flow.FlowStep (BaseFlowStep.java:logInfo(834)) - [max] tracking url: http://localhost:8080/
2015-02-12 16:12:49,250 INFO  mapred.LocalJobRunner (LocalJobRunner.java:runTasks(448)) - Waiting for map tasks
2015-02-12 16:12:49,251 INFO  mapred.LocalJobRunner (LocalJobRunner.java:run(224)) - Starting task: attempt_local1579825300_0013_m_000000_0
2015-02-12 16:12:49,251 INFO  util.ProcfsBasedProcessTree (ProcfsBasedProcessTree.java:isAvailable(181)) - ProcfsBasedProcessTree currently is supported only on Linux.
2015-02-12 16:12:49,252 INFO  mapred.Task (Task.java:initialize(587)) -  Using ResourceCalculatorProcessTree : null
2015-02-12 16:12:49,253 INFO  io.MultiInputSplit (MultiInputSplit.java:readFields(161)) - current split input path: file:/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/classes/test/data/lhs.txt
2015-02-12 16:12:49,253 INFO  mapred.MapTask (MapTask.java:updateJobWithSplit(482)) - Processing split: cascading.tap.hadoop.io.MultiInputSplit@67825189
2015-02-12 16:12:49,254 INFO  mapred.MapTask (MapTask.java:runOldMapper(436)) - numReduceTasks: 1
2015-02-12 16:12:49,270 INFO  mapred.MapTask (MapTask.java:setEquator(1202)) - (EQUATOR) 0 kvi 26214396(104857584)
2015-02-12 16:12:49,271 INFO  mapred.MapTask (MapTask.java:init(995)) - mapreduce.task.io.sort.mb: 100
2015-02-12 16:12:49,271 INFO  mapred.MapTask (MapTask.java:init(996)) - soft limit at 83886080
2015-02-12 16:12:49,271 INFO  mapred.MapTask (MapTask.java:init(997)) - bufstart = 0; bufvoid = 104857600
2015-02-12 16:12:49,271 INFO  mapred.MapTask (MapTask.java:init(998)) - kvstart = 26214396; length = 6553600
2015-02-12 16:12:49,272 INFO  mapred.MapTask (MapTask.java:createSortingCollector(402)) - Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer
2015-02-12 16:12:49,272 INFO  hadoop.FlowMapper (FlowMapper.java:configure(71)) - cascading version: 2.6.3
2015-02-12 16:12:49,272 INFO  hadoop.FlowMapper (FlowMapper.java:configure(72)) - child jvm opts: -Xmx200m
2015-02-12 16:12:49,277 INFO  hadoop.FlowMapper (FlowMapper.java:configure(87)) - sourcing from: Hfs["TextDelimited[['num', 'char']]"]["/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/classes/test/data/lhs.txt"]
2015-02-12 16:12:49,277 INFO  hadoop.FlowMapper (FlowMapper.java:configure(90)) - sinking to: GroupBy(max)[by:[{1}:'char']]
2015-02-12 16:12:49,277 INFO  assembly.AggregateBy (AggregateBy.java:prepare(267)) - using threshold value: 2
2015-02-12 16:12:49,278 INFO  assembly.AggregateBy (AggregateBy.java:removeEldestEntry(327)) - flushed keys num times: 1, with threshold: 2
2015-02-12 16:12:49,278 INFO  assembly.AggregateBy (AggregateBy.java:removeEldestEntry(328)) - mem on flush (mb), free: 199, total: 509, max: 672
2015-02-12 16:12:49,278 INFO  assembly.AggregateBy (AggregateBy.java:removeEldestEntry(333)) - total mem is 75% of max mem, to better utilize unused memory consider increasing current LRU threshold with system property "cascading.aggregateby.threshold"
2015-02-12 16:12:49,279 INFO  assembly.AggregateBy (AggregateBy.java:removeEldestEntry(327)) - flushed keys num times: 3, with threshold: 2
2015-02-12 16:12:49,279 INFO  assembly.AggregateBy (AggregateBy.java:removeEldestEntry(328)) - mem on flush (mb), free: 199, total: 509, max: 672
2015-02-12 16:12:49,279 INFO  assembly.AggregateBy (AggregateBy.java:removeEldestEntry(333)) - total mem is 75% of max mem, to better utilize unused memory consider increasing current LRU threshold with system property "cascading.aggregateby.threshold"
2015-02-12 16:12:49,280 INFO  assembly.AggregateBy (AggregateBy.java:removeEldestEntry(327)) - flushed keys num times: 5, with threshold: 2
2015-02-12 16:12:49,280 INFO  assembly.AggregateBy (AggregateBy.java:removeEldestEntry(328)) - mem on flush (mb), free: 199, total: 509, max: 672
2015-02-12 16:12:49,280 INFO  assembly.AggregateBy (AggregateBy.java:removeEldestEntry(333)) - total mem is 75% of max mem, to better utilize unused memory consider increasing current LRU threshold with system property "cascading.aggregateby.threshold"
2015-02-12 16:12:49,280 INFO  assembly.AggregateBy (AggregateBy.java:removeEldestEntry(327)) - flushed keys num times: 7, with threshold: 2
2015-02-12 16:12:49,280 INFO  assembly.AggregateBy (AggregateBy.java:removeEldestEntry(328)) - mem on flush (mb), free: 199, total: 509, max: 672
2015-02-12 16:12:49,280 INFO  assembly.AggregateBy (AggregateBy.java:removeEldestEntry(333)) - total mem is 75% of max mem, to better utilize unused memory consider increasing current LRU threshold with system property "cascading.aggregateby.threshold"
2015-02-12 16:12:49,281 INFO  mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) - 
2015-02-12 16:12:49,281 INFO  mapred.MapTask (MapTask.java:flush(1457)) - Starting flush of map output
2015-02-12 16:12:49,281 INFO  mapred.MapTask (MapTask.java:flush(1475)) - Spilling map output
2015-02-12 16:12:49,282 INFO  mapred.MapTask (MapTask.java:flush(1476)) - bufstart = 0; bufend = 135; bufvoid = 104857600
2015-02-12 16:12:49,282 INFO  mapred.MapTask (MapTask.java:flush(1478)) - kvstart = 26214396(104857584); kvend = 26214364(104857456); length = 33/6553600
2015-02-12 16:12:49,283 INFO  mapred.MapTask (MapTask.java:sortAndSpill(1660)) - Finished spill 0
2015-02-12 16:12:49,285 INFO  mapred.Task (Task.java:done(1001)) - Task:attempt_local1579825300_0013_m_000000_0 is done. And is in the process of committing
2015-02-12 16:12:49,286 INFO  mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) - file:/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/classes/test/data/lhs.txt:0+52
2015-02-12 16:12:49,286 INFO  mapred.Task (Task.java:sendDone(1121)) - Task 'attempt_local1579825300_0013_m_000000_0' done.
2015-02-12 16:12:49,286 INFO  mapred.LocalJobRunner (LocalJobRunner.java:run(249)) - Finishing task: attempt_local1579825300_0013_m_000000_0
2015-02-12 16:12:49,286 INFO  mapred.LocalJobRunner (LocalJobRunner.java:runTasks(456)) - map task executor complete.
2015-02-12 16:12:49,287 INFO  mapred.LocalJobRunner (LocalJobRunner.java:runTasks(448)) - Waiting for reduce tasks
2015-02-12 16:12:49,287 INFO  mapred.LocalJobRunner (LocalJobRunner.java:run(302)) - Starting task: attempt_local1579825300_0013_r_000000_0
2015-02-12 16:12:49,288 INFO  util.ProcfsBasedProcessTree (ProcfsBasedProcessTree.java:isAvailable(181)) - ProcfsBasedProcessTree currently is supported only on Linux.
2015-02-12 16:12:49,288 INFO  mapred.Task (Task.java:initialize(587)) -  Using ResourceCalculatorProcessTree : null
2015-02-12 16:12:49,288 INFO  mapred.ReduceTask (ReduceTask.java:run(362)) - Using ShuffleConsumerPlugin: org.apache.hadoop.mapreduce.task.reduce.Shuffle@13233788
2015-02-12 16:12:49,289 INFO  reduce.MergeManagerImpl (MergeManagerImpl.java:<init>(195)) - MergerManager: memoryLimit=493250144, maxSingleShuffleLimit=123312536, mergeThreshold=325545120, ioSortFactor=10, memToMemMergeOutputsThreshold=10
2015-02-12 16:12:49,289 INFO  reduce.EventFetcher (EventFetcher.java:run(61)) - attempt_local1579825300_0013_r_000000_0 Thread started: EventFetcher for fetching Map Completion Events
2015-02-12 16:12:49,290 INFO  reduce.LocalFetcher (LocalFetcher.java:copyMapOutput(141)) - localfetcher#11 about to shuffle output of map attempt_local1579825300_0013_m_000000_0 decomp: 155 len: 159 to MEMORY
2015-02-12 16:12:49,291 INFO  reduce.InMemoryMapOutput (InMemoryMapOutput.java:shuffle(100)) - Read 155 bytes from map-output for attempt_local1579825300_0013_m_000000_0
2015-02-12 16:12:49,291 INFO  reduce.MergeManagerImpl (MergeManagerImpl.java:closeInMemoryFile(313)) - closeInMemoryFile -> map-output of size: 155, inMemoryMapOutputs.size() -> 1, commitMemory -> 0, usedMemory ->155
2015-02-12 16:12:49,291 INFO  reduce.EventFetcher (EventFetcher.java:run(76)) - EventFetcher is interrupted.. Returning
2015-02-12 16:12:49,292 INFO  mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) - 1 / 1 copied.
2015-02-12 16:12:49,292 INFO  reduce.MergeManagerImpl (MergeManagerImpl.java:finalMerge(673)) - finalMerge called with 1 in-memory map-outputs and 0 on-disk map-outputs
2015-02-12 16:12:49,293 INFO  mapred.Merger (Merger.java:merge(606)) - Merging 1 sorted segments
2015-02-12 16:12:49,293 INFO  mapred.Merger (Merger.java:merge(705)) - Down to the last merge-pass, with 1 segments left of total size: 146 bytes
2015-02-12 16:12:49,294 INFO  reduce.MergeManagerImpl (MergeManagerImpl.java:finalMerge(750)) - Merged 1 segments, 155 bytes to disk to satisfy reduce memory limit
2015-02-12 16:12:49,294 INFO  reduce.MergeManagerImpl (MergeManagerImpl.java:finalMerge(780)) - Merging 1 files, 159 bytes from disk
2015-02-12 16:12:49,294 INFO  reduce.MergeManagerImpl (MergeManagerImpl.java:finalMerge(795)) - Merging 0 segments, 0 bytes from memory into reduce
2015-02-12 16:12:49,295 INFO  mapred.Merger (Merger.java:merge(606)) - Merging 1 sorted segments
2015-02-12 16:12:49,295 INFO  mapred.Merger (Merger.java:merge(705)) - Down to the last merge-pass, with 1 segments left of total size: 146 bytes
2015-02-12 16:12:49,295 INFO  mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) - 1 / 1 copied.
2015-02-12 16:12:49,296 INFO  hadoop.FlowReducer (FlowReducer.java:configure(78)) - cascading version: 2.6.3
2015-02-12 16:12:49,296 INFO  hadoop.FlowReducer (FlowReducer.java:configure(79)) - child jvm opts: -Xmx200m
2015-02-12 16:12:49,301 INFO  hadoop.FlowReducer (FlowReducer.java:configure(97)) - sourcing from: GroupBy(max)[by:[{1}:'char']]
2015-02-12 16:12:49,301 INFO  hadoop.FlowReducer (FlowReducer.java:configure(100)) - sinking to: Hfs["TextDelimited[['char', 'max']]"]["/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/test/output/hadoop2-mr1/assemblyhelpersplatform/maxby"]
2015-02-12 16:12:49,309 INFO  mapred.Task (Task.java:done(1001)) - Task:attempt_local1579825300_0013_r_000000_0 is done. And is in the process of committing
2015-02-12 16:12:49,310 INFO  mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) - 1 / 1 copied.
2015-02-12 16:12:49,310 INFO  mapred.Task (Task.java:commit(1162)) - Task attempt_local1579825300_0013_r_000000_0 is allowed to commit now
2015-02-12 16:12:49,310 INFO  output.FileOutputCommitter (FileOutputCommitter.java:commitTask(439)) - Saved output of task 'attempt_local1579825300_0013_r_000000_0' to file:/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/test/output/hadoop2-mr1/assemblyhelpersplatform/maxby/_temporary/0/task_local1579825300_0013_r_000000
2015-02-12 16:12:49,311 INFO  mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) - reduce > reduce
2015-02-12 16:12:49,311 INFO  mapred.Task (Task.java:sendDone(1121)) - Task 'attempt_local1579825300_0013_r_000000_0' done.
2015-02-12 16:12:49,311 INFO  mapred.LocalJobRunner (LocalJobRunner.java:run(325)) - Finishing task: attempt_local1579825300_0013_r_000000_0
2015-02-12 16:12:49,312 INFO  mapred.LocalJobRunner (LocalJobRunner.java:runTasks(456)) - reduce task executor complete.
2015-02-12 16:12:49,322 INFO  util.Hadoop18TapUtil (Hadoop18TapUtil.java:cleanTempPath(219)) - deleting temp path /Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/test/output/hadoop2-mr1/assemblyhelpersplatform/maxby/_temporary
2015-02-12 16:12:49,343 INFO  mapred.FileInputFormat (FileInputFormat.java:listStatus(247)) - Total input paths to process : 1
2015-02-12 16:12:49,364 INFO  mapred.FileInputFormat (FileInputFormat.java:listStatus(247)) - Total input paths to process : 1
2015-02-12 16:12:49,365 INFO  cascading.PlatformTestCase (PlatformTestCase.java:tearDown(209)) - copying to local /Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/test/output//hadoop2-mr1/assemblyhelpersplatform/maxby
2015-02-12 16:12:49,377 INFO  util.HadoopUtil (HadoopUtil.java:findMainClass(336)) - using default application jar, may cause class not found exceptions on the cluster
2015-02-12 16:12:49,377 INFO  planner.HadoopPlanner (HadoopPlanner.java:initialize(225)) - using application jar: /Users/soberoi/.gradle/caches/modules-2/files-2.1/cascading/cascading-hadoop2-mr1/2.6.3/9ae016e2b8d246b9c1de9168e715d9f1b971aa4a/cascading-hadoop2-mr1-2.6.3.jar
2015-02-12 16:12:49,382 INFO  Configuration.deprecation (Configuration.java:warnOnceIfDeprecated(1022)) - mapred.used.genericoptionsparser is deprecated. Instead, use mapreduce.client.genericoptionsparser.used
2015-02-12 16:12:49,383 INFO  Configuration.deprecation (Configuration.java:warnOnceIfDeprecated(1022)) - mapred.job.tracker is deprecated. Instead, use mapreduce.jobtracker.address
2015-02-12 16:12:49,383 INFO  Configuration.deprecation (Configuration.java:warnOnceIfDeprecated(1022)) - mapred.output.compress is deprecated. Instead, use mapreduce.output.fileoutputformat.compress
2015-02-12 16:12:49,388 INFO  flow.Flow (BaseFlow.java:logInfo(1378)) - [min] starting
2015-02-12 16:12:49,388 INFO  flow.Flow (BaseFlow.java:logInfo(1378)) - [min]  source: Hfs["TextDelimited[['num', 'char']]"]["/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/classes/test/data/lhs.txt"]
2015-02-12 16:12:49,388 INFO  flow.Flow (BaseFlow.java:logInfo(1378)) - [min]  sink: Hfs["TextDelimited[['char', 'min']]"]["/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/test/output/hadoop2-mr1/assemblyhelpersplatform/minby"]
2015-02-12 16:12:49,388 INFO  flow.Flow (BaseFlow.java:logInfo(1378)) - [min]  parallel execution is enabled: false
2015-02-12 16:12:49,388 INFO  flow.Flow (BaseFlow.java:logInfo(1378)) - [min]  starting jobs: 1
2015-02-12 16:12:49,388 INFO  flow.Flow (BaseFlow.java:logInfo(1378)) - [min]  allocating threads: 1
2015-02-12 16:12:49,388 INFO  flow.FlowStep (BaseFlowStep.java:logInfo(834)) - [min] starting step: (1/1) ...mblyhelpersplatform/minby
2015-02-12 16:12:49,389 INFO  jvm.JvmMetrics (JvmMetrics.java:init(71)) - Cannot initialize JVM Metrics with processName=JobTracker, sessionId= - already initialized
2015-02-12 16:12:49,389 INFO  jvm.JvmMetrics (JvmMetrics.java:init(71)) - Cannot initialize JVM Metrics with processName=JobTracker, sessionId= - already initialized
2015-02-12 16:12:49,407 INFO  mapred.FileInputFormat (FileInputFormat.java:listStatus(247)) - Total input paths to process : 1
2015-02-12 16:12:49,423 INFO  mapreduce.JobSubmitter (JobSubmitter.java:submitJobInternal(492)) - number of splits:1
2015-02-12 16:12:49,434 INFO  mapreduce.JobSubmitter (JobSubmitter.java:printTokens(575)) - Submitting tokens for job: job_local1907556576_0014
2015-02-12 16:12:49,444 WARN  conf.Configuration (Configuration.java:loadProperty(2510)) - file:/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/tmp/cascading/staging/soberoi1907556576/.staging/job_local1907556576_0014/job.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.retry.interval;  Ignoring.
2015-02-12 16:12:49,445 WARN  conf.Configuration (Configuration.java:loadProperty(2510)) - file:/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/tmp/cascading/staging/soberoi1907556576/.staging/job_local1907556576_0014/job.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.attempts;  Ignoring.
2015-02-12 16:12:49,476 WARN  conf.Configuration (Configuration.java:loadProperty(2510)) - file:/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/test/tmp/mapred/local/localRunner/soberoi/job_local1907556576_0014/job_local1907556576_0014.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.retry.interval;  Ignoring.
2015-02-12 16:12:49,477 WARN  conf.Configuration (Configuration.java:loadProperty(2510)) - file:/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/test/tmp/mapred/local/localRunner/soberoi/job_local1907556576_0014/job_local1907556576_0014.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.attempts;  Ignoring.
2015-02-12 16:12:49,477 INFO  mapreduce.Job (Job.java:submit(1299)) - The url to track the job: http://localhost:8080/
2015-02-12 16:12:49,478 INFO  mapred.LocalJobRunner (LocalJobRunner.java:createOutputCommitter(471)) - OutputCommitter set in config null
2015-02-12 16:12:49,478 INFO  flow.FlowStep (BaseFlowStep.java:logInfo(834)) - [min] submitted hadoop job: job_local1907556576_0014
2015-02-12 16:12:49,478 INFO  mapred.LocalJobRunner (LocalJobRunner.java:createOutputCommitter(489)) - OutputCommitter is org.apache.hadoop.mapred.FileOutputCommitter
2015-02-12 16:12:49,478 INFO  flow.FlowStep (BaseFlowStep.java:logInfo(834)) - [min] tracking url: http://localhost:8080/
2015-02-12 16:12:49,480 INFO  mapred.LocalJobRunner (LocalJobRunner.java:runTasks(448)) - Waiting for map tasks
2015-02-12 16:12:49,480 INFO  mapred.LocalJobRunner (LocalJobRunner.java:run(224)) - Starting task: attempt_local1907556576_0014_m_000000_0
2015-02-12 16:12:49,481 INFO  util.ProcfsBasedProcessTree (ProcfsBasedProcessTree.java:isAvailable(181)) - ProcfsBasedProcessTree currently is supported only on Linux.
2015-02-12 16:12:49,481 INFO  mapred.Task (Task.java:initialize(587)) -  Using ResourceCalculatorProcessTree : null
2015-02-12 16:12:49,482 INFO  io.MultiInputSplit (MultiInputSplit.java:readFields(161)) - current split input path: file:/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/classes/test/data/lhs.txt
2015-02-12 16:12:49,483 INFO  mapred.MapTask (MapTask.java:updateJobWithSplit(482)) - Processing split: cascading.tap.hadoop.io.MultiInputSplit@367019ee
2015-02-12 16:12:49,484 INFO  mapred.MapTask (MapTask.java:runOldMapper(436)) - numReduceTasks: 1
2015-02-12 16:12:49,499 INFO  mapred.MapTask (MapTask.java:setEquator(1202)) - (EQUATOR) 0 kvi 26214396(104857584)
2015-02-12 16:12:49,500 INFO  mapred.MapTask (MapTask.java:init(995)) - mapreduce.task.io.sort.mb: 100
2015-02-12 16:12:49,500 INFO  mapred.MapTask (MapTask.java:init(996)) - soft limit at 83886080
2015-02-12 16:12:49,500 INFO  mapred.MapTask (MapTask.java:init(997)) - bufstart = 0; bufvoid = 104857600
2015-02-12 16:12:49,500 INFO  mapred.MapTask (MapTask.java:init(998)) - kvstart = 26214396; length = 6553600
2015-02-12 16:12:49,502 INFO  mapred.MapTask (MapTask.java:createSortingCollector(402)) - Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer
2015-02-12 16:12:49,502 INFO  hadoop.FlowMapper (FlowMapper.java:configure(71)) - cascading version: 2.6.3
2015-02-12 16:12:49,502 INFO  hadoop.FlowMapper (FlowMapper.java:configure(72)) - child jvm opts: -Xmx200m
2015-02-12 16:12:49,509 INFO  hadoop.FlowMapper (FlowMapper.java:configure(87)) - sourcing from: Hfs["TextDelimited[['num', 'char']]"]["/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/classes/test/data/lhs.txt"]
2015-02-12 16:12:49,509 INFO  hadoop.FlowMapper (FlowMapper.java:configure(90)) - sinking to: GroupBy(min)[by:[{1}:'char']]
2015-02-12 16:12:49,510 INFO  assembly.AggregateBy (AggregateBy.java:prepare(267)) - using threshold value: 2
2015-02-12 16:12:49,511 INFO  assembly.AggregateBy (AggregateBy.java:removeEldestEntry(327)) - flushed keys num times: 1, with threshold: 2
2015-02-12 16:12:49,511 INFO  assembly.AggregateBy (AggregateBy.java:removeEldestEntry(328)) - mem on flush (mb), free: 187, total: 498, max: 672
2015-02-12 16:12:49,511 INFO  assembly.AggregateBy (AggregateBy.java:removeEldestEntry(333)) - total mem is 74% of max mem, to better utilize unused memory consider increasing current LRU threshold with system property "cascading.aggregateby.threshold"
2015-02-12 16:12:49,512 INFO  assembly.AggregateBy (AggregateBy.java:removeEldestEntry(327)) - flushed keys num times: 3, with threshold: 2
2015-02-12 16:12:49,512 INFO  assembly.AggregateBy (AggregateBy.java:removeEldestEntry(328)) - mem on flush (mb), free: 187, total: 498, max: 672
2015-02-12 16:12:49,512 INFO  assembly.AggregateBy (AggregateBy.java:removeEldestEntry(333)) - total mem is 74% of max mem, to better utilize unused memory consider increasing current LRU threshold with system property "cascading.aggregateby.threshold"
2015-02-12 16:12:49,512 INFO  assembly.AggregateBy (AggregateBy.java:removeEldestEntry(327)) - flushed keys num times: 5, with threshold: 2
2015-02-12 16:12:49,513 INFO  assembly.AggregateBy (AggregateBy.java:removeEldestEntry(328)) - mem on flush (mb), free: 187, total: 498, max: 672
2015-02-12 16:12:49,513 INFO  assembly.AggregateBy (AggregateBy.java:removeEldestEntry(333)) - total mem is 74% of max mem, to better utilize unused memory consider increasing current LRU threshold with system property "cascading.aggregateby.threshold"
2015-02-12 16:12:49,513 INFO  assembly.AggregateBy (AggregateBy.java:removeEldestEntry(327)) - flushed keys num times: 7, with threshold: 2
2015-02-12 16:12:49,513 INFO  assembly.AggregateBy (AggregateBy.java:removeEldestEntry(328)) - mem on flush (mb), free: 187, total: 498, max: 672
2015-02-12 16:12:49,513 INFO  assembly.AggregateBy (AggregateBy.java:removeEldestEntry(333)) - total mem is 74% of max mem, to better utilize unused memory consider increasing current LRU threshold with system property "cascading.aggregateby.threshold"
2015-02-12 16:12:49,514 INFO  mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) - 
2015-02-12 16:12:49,514 INFO  mapred.MapTask (MapTask.java:flush(1457)) - Starting flush of map output
2015-02-12 16:12:49,514 INFO  mapred.MapTask (MapTask.java:flush(1475)) - Spilling map output
2015-02-12 16:12:49,515 INFO  mapred.MapTask (MapTask.java:flush(1476)) - bufstart = 0; bufend = 135; bufvoid = 104857600
2015-02-12 16:12:49,515 INFO  mapred.MapTask (MapTask.java:flush(1478)) - kvstart = 26214396(104857584); kvend = 26214364(104857456); length = 33/6553600
2015-02-12 16:12:49,517 INFO  mapred.MapTask (MapTask.java:sortAndSpill(1660)) - Finished spill 0
2015-02-12 16:12:49,518 INFO  mapred.Task (Task.java:done(1001)) - Task:attempt_local1907556576_0014_m_000000_0 is done. And is in the process of committing
2015-02-12 16:12:49,520 INFO  mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) - file:/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/classes/test/data/lhs.txt:0+52
2015-02-12 16:12:49,520 INFO  mapred.Task (Task.java:sendDone(1121)) - Task 'attempt_local1907556576_0014_m_000000_0' done.
2015-02-12 16:12:49,520 INFO  mapred.LocalJobRunner (LocalJobRunner.java:run(249)) - Finishing task: attempt_local1907556576_0014_m_000000_0
2015-02-12 16:12:49,520 INFO  mapred.LocalJobRunner (LocalJobRunner.java:runTasks(456)) - map task executor complete.
2015-02-12 16:12:49,521 INFO  mapred.LocalJobRunner (LocalJobRunner.java:runTasks(448)) - Waiting for reduce tasks
2015-02-12 16:12:49,521 INFO  mapred.LocalJobRunner (LocalJobRunner.java:run(302)) - Starting task: attempt_local1907556576_0014_r_000000_0
2015-02-12 16:12:49,522 INFO  util.ProcfsBasedProcessTree (ProcfsBasedProcessTree.java:isAvailable(181)) - ProcfsBasedProcessTree currently is supported only on Linux.
2015-02-12 16:12:49,523 INFO  mapred.Task (Task.java:initialize(587)) -  Using ResourceCalculatorProcessTree : null
2015-02-12 16:12:49,523 INFO  mapred.ReduceTask (ReduceTask.java:run(362)) - Using ShuffleConsumerPlugin: org.apache.hadoop.mapreduce.task.reduce.Shuffle@45d2d12f
2015-02-12 16:12:49,523 INFO  reduce.MergeManagerImpl (MergeManagerImpl.java:<init>(195)) - MergerManager: memoryLimit=493250144, maxSingleShuffleLimit=123312536, mergeThreshold=325545120, ioSortFactor=10, memToMemMergeOutputsThreshold=10
2015-02-12 16:12:49,524 INFO  reduce.EventFetcher (EventFetcher.java:run(61)) - attempt_local1907556576_0014_r_000000_0 Thread started: EventFetcher for fetching Map Completion Events
2015-02-12 16:12:49,525 INFO  reduce.LocalFetcher (LocalFetcher.java:copyMapOutput(141)) - localfetcher#12 about to shuffle output of map attempt_local1907556576_0014_m_000000_0 decomp: 155 len: 159 to MEMORY
2015-02-12 16:12:49,526 INFO  reduce.InMemoryMapOutput (InMemoryMapOutput.java:shuffle(100)) - Read 155 bytes from map-output for attempt_local1907556576_0014_m_000000_0
2015-02-12 16:12:49,526 INFO  reduce.MergeManagerImpl (MergeManagerImpl.java:closeInMemoryFile(313)) - closeInMemoryFile -> map-output of size: 155, inMemoryMapOutputs.size() -> 1, commitMemory -> 0, usedMemory ->155
2015-02-12 16:12:49,526 INFO  reduce.EventFetcher (EventFetcher.java:run(76)) - EventFetcher is interrupted.. Returning
2015-02-12 16:12:49,527 INFO  mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) - 1 / 1 copied.
2015-02-12 16:12:49,527 INFO  reduce.MergeManagerImpl (MergeManagerImpl.java:finalMerge(673)) - finalMerge called with 1 in-memory map-outputs and 0 on-disk map-outputs
2015-02-12 16:12:49,529 INFO  mapred.Merger (Merger.java:merge(606)) - Merging 1 sorted segments
2015-02-12 16:12:49,529 INFO  mapred.Merger (Merger.java:merge(705)) - Down to the last merge-pass, with 1 segments left of total size: 146 bytes
2015-02-12 16:12:49,530 INFO  reduce.MergeManagerImpl (MergeManagerImpl.java:finalMerge(750)) - Merged 1 segments, 155 bytes to disk to satisfy reduce memory limit
2015-02-12 16:12:49,531 INFO  reduce.MergeManagerImpl (MergeManagerImpl.java:finalMerge(780)) - Merging 1 files, 159 bytes from disk
2015-02-12 16:12:49,531 INFO  reduce.MergeManagerImpl (MergeManagerImpl.java:finalMerge(795)) - Merging 0 segments, 0 bytes from memory into reduce
2015-02-12 16:12:49,531 INFO  mapred.Merger (Merger.java:merge(606)) - Merging 1 sorted segments
2015-02-12 16:12:49,531 INFO  mapred.Merger (Merger.java:merge(705)) - Down to the last merge-pass, with 1 segments left of total size: 146 bytes
2015-02-12 16:12:49,532 INFO  mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) - 1 / 1 copied.
2015-02-12 16:12:49,532 INFO  hadoop.FlowReducer (FlowReducer.java:configure(78)) - cascading version: 2.6.3
2015-02-12 16:12:49,532 INFO  hadoop.FlowReducer (FlowReducer.java:configure(79)) - child jvm opts: -Xmx200m
2015-02-12 16:12:49,539 INFO  hadoop.FlowReducer (FlowReducer.java:configure(97)) - sourcing from: GroupBy(min)[by:[{1}:'char']]
2015-02-12 16:12:49,539 INFO  hadoop.FlowReducer (FlowReducer.java:configure(100)) - sinking to: Hfs["TextDelimited[['char', 'min']]"]["/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/test/output/hadoop2-mr1/assemblyhelpersplatform/minby"]
2015-02-12 16:12:49,549 INFO  mapred.Task (Task.java:done(1001)) - Task:attempt_local1907556576_0014_r_000000_0 is done. And is in the process of committing
2015-02-12 16:12:49,550 INFO  mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) - 1 / 1 copied.
2015-02-12 16:12:49,551 INFO  mapred.Task (Task.java:commit(1162)) - Task attempt_local1907556576_0014_r_000000_0 is allowed to commit now
2015-02-12 16:12:49,551 INFO  output.FileOutputCommitter (FileOutputCommitter.java:commitTask(439)) - Saved output of task 'attempt_local1907556576_0014_r_000000_0' to file:/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/test/output/hadoop2-mr1/assemblyhelpersplatform/minby/_temporary/0/task_local1907556576_0014_r_000000
2015-02-12 16:12:49,552 INFO  mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) - reduce > reduce
2015-02-12 16:12:49,552 INFO  mapred.Task (Task.java:sendDone(1121)) - Task 'attempt_local1907556576_0014_r_000000_0' done.
2015-02-12 16:12:49,553 INFO  mapred.LocalJobRunner (LocalJobRunner.java:run(325)) - Finishing task: attempt_local1907556576_0014_r_000000_0
2015-02-12 16:12:49,553 INFO  mapred.LocalJobRunner (LocalJobRunner.java:runTasks(456)) - reduce task executor complete.
2015-02-12 16:12:49,573 INFO  util.Hadoop18TapUtil (Hadoop18TapUtil.java:cleanTempPath(219)) - deleting temp path /Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/test/output/hadoop2-mr1/assemblyhelpersplatform/minby/_temporary
2015-02-12 16:12:49,605 INFO  mapred.FileInputFormat (FileInputFormat.java:listStatus(247)) - Total input paths to process : 1
2015-02-12 16:12:49,637 INFO  mapred.FileInputFormat (FileInputFormat.java:listStatus(247)) - Total input paths to process : 1
2015-02-12 16:12:49,638 INFO  cascading.PlatformTestCase (PlatformTestCase.java:tearDown(209)) - copying to local /Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/test/output//hadoop2-mr1/assemblyhelpersplatform/minby
2015-02-12 16:12:49,655 INFO  util.HadoopUtil (HadoopUtil.java:findMainClass(336)) - using default application jar, may cause class not found exceptions on the cluster
2015-02-12 16:12:49,655 INFO  planner.HadoopPlanner (HadoopPlanner.java:initialize(225)) - using application jar: /Users/soberoi/.gradle/caches/modules-2/files-2.1/cascading/cascading-hadoop2-mr1/2.6.3/9ae016e2b8d246b9c1de9168e715d9f1b971aa4a/cascading-hadoop2-mr1-2.6.3.jar
2015-02-12 16:12:49,659 INFO  Configuration.deprecation (Configuration.java:warnOnceIfDeprecated(1022)) - mapred.used.genericoptionsparser is deprecated. Instead, use mapreduce.client.genericoptionsparser.used
2015-02-12 16:12:49,661 INFO  Configuration.deprecation (Configuration.java:warnOnceIfDeprecated(1022)) - mapred.job.tracker is deprecated. Instead, use mapreduce.jobtracker.address
2015-02-12 16:12:49,661 INFO  Configuration.deprecation (Configuration.java:warnOnceIfDeprecated(1022)) - mapred.output.compress is deprecated. Instead, use mapreduce.output.fileoutputformat.compress
2015-02-12 16:12:49,666 INFO  flow.Flow (BaseFlow.java:logInfo(1378)) - [sum] starting
2015-02-12 16:12:49,667 INFO  flow.Flow (BaseFlow.java:logInfo(1378)) - [sum]  source: Hfs["TextDelimited[['num', 'char']]"]["/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/classes/test/data/lhs.txt"]
2015-02-12 16:12:49,667 INFO  flow.Flow (BaseFlow.java:logInfo(1378)) - [sum]  sink: Hfs["TextDelimited[['char', 'sum']]"]["/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/test/output/hadoop2-mr1/assemblyhelpersplatform/sum"]
2015-02-12 16:12:49,667 INFO  flow.Flow (BaseFlow.java:logInfo(1378)) - [sum]  parallel execution is enabled: false
2015-02-12 16:12:49,667 INFO  flow.Flow (BaseFlow.java:logInfo(1378)) - [sum]  starting jobs: 1
2015-02-12 16:12:49,667 INFO  flow.Flow (BaseFlow.java:logInfo(1378)) - [sum]  allocating threads: 1
2015-02-12 16:12:49,668 INFO  flow.FlowStep (BaseFlowStep.java:logInfo(834)) - [sum] starting step: (1/1) ...semblyhelpersplatform/sum
2015-02-12 16:12:49,668 INFO  jvm.JvmMetrics (JvmMetrics.java:init(71)) - Cannot initialize JVM Metrics with processName=JobTracker, sessionId= - already initialized
2015-02-12 16:12:49,669 INFO  jvm.JvmMetrics (JvmMetrics.java:init(71)) - Cannot initialize JVM Metrics with processName=JobTracker, sessionId= - already initialized
2015-02-12 16:12:49,689 INFO  mapred.FileInputFormat (FileInputFormat.java:listStatus(247)) - Total input paths to process : 1
2015-02-12 16:12:49,705 INFO  mapreduce.JobSubmitter (JobSubmitter.java:submitJobInternal(492)) - number of splits:1
2015-02-12 16:12:49,716 INFO  mapreduce.JobSubmitter (JobSubmitter.java:printTokens(575)) - Submitting tokens for job: job_local1346937241_0015
2015-02-12 16:12:49,726 WARN  conf.Configuration (Configuration.java:loadProperty(2510)) - file:/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/tmp/cascading/staging/soberoi1346937241/.staging/job_local1346937241_0015/job.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.retry.interval;  Ignoring.
2015-02-12 16:12:49,727 WARN  conf.Configuration (Configuration.java:loadProperty(2510)) - file:/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/tmp/cascading/staging/soberoi1346937241/.staging/job_local1346937241_0015/job.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.attempts;  Ignoring.
2015-02-12 16:12:49,757 WARN  conf.Configuration (Configuration.java:loadProperty(2510)) - file:/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/test/tmp/mapred/local/localRunner/soberoi/job_local1346937241_0015/job_local1346937241_0015.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.retry.interval;  Ignoring.
2015-02-12 16:12:49,758 WARN  conf.Configuration (Configuration.java:loadProperty(2510)) - file:/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/test/tmp/mapred/local/localRunner/soberoi/job_local1346937241_0015/job_local1346937241_0015.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.attempts;  Ignoring.
2015-02-12 16:12:49,759 INFO  mapreduce.Job (Job.java:submit(1299)) - The url to track the job: http://localhost:8080/
2015-02-12 16:12:49,759 INFO  mapred.LocalJobRunner (LocalJobRunner.java:createOutputCommitter(471)) - OutputCommitter set in config null
2015-02-12 16:12:49,759 INFO  flow.FlowStep (BaseFlowStep.java:logInfo(834)) - [sum] submitted hadoop job: job_local1346937241_0015
2015-02-12 16:12:49,759 INFO  mapred.LocalJobRunner (LocalJobRunner.java:createOutputCommitter(489)) - OutputCommitter is org.apache.hadoop.mapred.FileOutputCommitter
2015-02-12 16:12:49,759 INFO  flow.FlowStep (BaseFlowStep.java:logInfo(834)) - [sum] tracking url: http://localhost:8080/
2015-02-12 16:12:49,761 INFO  mapred.LocalJobRunner (LocalJobRunner.java:runTasks(448)) - Waiting for map tasks
2015-02-12 16:12:49,761 INFO  mapred.LocalJobRunner (LocalJobRunner.java:run(224)) - Starting task: attempt_local1346937241_0015_m_000000_0
2015-02-12 16:12:49,762 INFO  util.ProcfsBasedProcessTree (ProcfsBasedProcessTree.java:isAvailable(181)) - ProcfsBasedProcessTree currently is supported only on Linux.
2015-02-12 16:12:49,762 INFO  mapred.Task (Task.java:initialize(587)) -  Using ResourceCalculatorProcessTree : null
2015-02-12 16:12:49,763 INFO  io.MultiInputSplit (MultiInputSplit.java:readFields(161)) - current split input path: file:/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/classes/test/data/lhs.txt
2015-02-12 16:12:49,763 INFO  mapred.MapTask (MapTask.java:updateJobWithSplit(482)) - Processing split: cascading.tap.hadoop.io.MultiInputSplit@3e708bac
2015-02-12 16:12:49,764 INFO  mapred.MapTask (MapTask.java:runOldMapper(436)) - numReduceTasks: 1
2015-02-12 16:12:49,780 INFO  mapred.MapTask (MapTask.java:setEquator(1202)) - (EQUATOR) 0 kvi 26214396(104857584)
2015-02-12 16:12:49,780 INFO  mapred.MapTask (MapTask.java:init(995)) - mapreduce.task.io.sort.mb: 100
2015-02-12 16:12:49,780 INFO  mapred.MapTask (MapTask.java:init(996)) - soft limit at 83886080
2015-02-12 16:12:49,780 INFO  mapred.MapTask (MapTask.java:init(997)) - bufstart = 0; bufvoid = 104857600
2015-02-12 16:12:49,780 INFO  mapred.MapTask (MapTask.java:init(998)) - kvstart = 26214396; length = 6553600
2015-02-12 16:12:49,781 INFO  mapred.MapTask (MapTask.java:createSortingCollector(402)) - Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer
2015-02-12 16:12:49,781 INFO  hadoop.FlowMapper (FlowMapper.java:configure(71)) - cascading version: 2.6.3
2015-02-12 16:12:49,781 INFO  hadoop.FlowMapper (FlowMapper.java:configure(72)) - child jvm opts: -Xmx200m
2015-02-12 16:12:49,786 INFO  hadoop.FlowMapper (FlowMapper.java:configure(87)) - sourcing from: Hfs["TextDelimited[['num', 'char']]"]["/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/classes/test/data/lhs.txt"]
2015-02-12 16:12:49,786 INFO  hadoop.FlowMapper (FlowMapper.java:configure(90)) - sinking to: GroupBy(sum)[by:[{1}:'char']]
2015-02-12 16:12:49,786 INFO  assembly.AggregateBy (AggregateBy.java:prepare(267)) - using threshold value: 2
2015-02-12 16:12:49,787 INFO  assembly.AggregateBy (AggregateBy.java:removeEldestEntry(327)) - flushed keys num times: 1, with threshold: 2
2015-02-12 16:12:49,787 INFO  assembly.AggregateBy (AggregateBy.java:removeEldestEntry(328)) - mem on flush (mb), free: 197, total: 509, max: 672
2015-02-12 16:12:49,787 INFO  assembly.AggregateBy (AggregateBy.java:removeEldestEntry(333)) - total mem is 75% of max mem, to better utilize unused memory consider increasing current LRU threshold with system property "cascading.aggregateby.threshold"
2015-02-12 16:12:49,788 INFO  assembly.AggregateBy (AggregateBy.java:removeEldestEntry(327)) - flushed keys num times: 3, with threshold: 2
2015-02-12 16:12:49,788 INFO  assembly.AggregateBy (AggregateBy.java:removeEldestEntry(328)) - mem on flush (mb), free: 197, total: 509, max: 672
2015-02-12 16:12:49,788 INFO  assembly.AggregateBy (AggregateBy.java:removeEldestEntry(333)) - total mem is 75% of max mem, to better utilize unused memory consider increasing current LRU threshold with system property "cascading.aggregateby.threshold"
2015-02-12 16:12:49,788 INFO  assembly.AggregateBy (AggregateBy.java:removeEldestEntry(327)) - flushed keys num times: 5, with threshold: 2
2015-02-12 16:12:49,788 INFO  assembly.AggregateBy (AggregateBy.java:removeEldestEntry(328)) - mem on flush (mb), free: 197, total: 509, max: 672
2015-02-12 16:12:49,788 INFO  assembly.AggregateBy (AggregateBy.java:removeEldestEntry(333)) - total mem is 75% of max mem, to better utilize unused memory consider increasing current LRU threshold with system property "cascading.aggregateby.threshold"
2015-02-12 16:12:49,789 INFO  assembly.AggregateBy (AggregateBy.java:removeEldestEntry(327)) - flushed keys num times: 7, with threshold: 2
2015-02-12 16:12:49,789 INFO  assembly.AggregateBy (AggregateBy.java:removeEldestEntry(328)) - mem on flush (mb), free: 197, total: 509, max: 672
2015-02-12 16:12:49,789 INFO  assembly.AggregateBy (AggregateBy.java:removeEldestEntry(333)) - total mem is 75% of max mem, to better utilize unused memory consider increasing current LRU threshold with system property "cascading.aggregateby.threshold"
2015-02-12 16:12:49,790 INFO  mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) - 
2015-02-12 16:12:49,790 INFO  mapred.MapTask (MapTask.java:flush(1457)) - Starting flush of map output
2015-02-12 16:12:49,790 INFO  mapred.MapTask (MapTask.java:flush(1475)) - Spilling map output
2015-02-12 16:12:49,790 INFO  mapred.MapTask (MapTask.java:flush(1476)) - bufstart = 0; bufend = 99; bufvoid = 104857600
2015-02-12 16:12:49,790 INFO  mapred.MapTask (MapTask.java:flush(1478)) - kvstart = 26214396(104857584); kvend = 26214364(104857456); length = 33/6553600
2015-02-12 16:12:49,792 INFO  mapred.MapTask (MapTask.java:sortAndSpill(1660)) - Finished spill 0
2015-02-12 16:12:49,793 INFO  mapred.Task (Task.java:done(1001)) - Task:attempt_local1346937241_0015_m_000000_0 is done. And is in the process of committing
2015-02-12 16:12:49,794 INFO  mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) - file:/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/classes/test/data/lhs.txt:0+52
2015-02-12 16:12:49,794 INFO  mapred.Task (Task.java:sendDone(1121)) - Task 'attempt_local1346937241_0015_m_000000_0' done.
2015-02-12 16:12:49,795 INFO  mapred.LocalJobRunner (LocalJobRunner.java:run(249)) - Finishing task: attempt_local1346937241_0015_m_000000_0
2015-02-12 16:12:49,795 INFO  mapred.LocalJobRunner (LocalJobRunner.java:runTasks(456)) - map task executor complete.
2015-02-12 16:12:49,795 INFO  mapred.LocalJobRunner (LocalJobRunner.java:runTasks(448)) - Waiting for reduce tasks
2015-02-12 16:12:49,795 INFO  mapred.LocalJobRunner (LocalJobRunner.java:run(302)) - Starting task: attempt_local1346937241_0015_r_000000_0
2015-02-12 16:12:49,796 INFO  util.ProcfsBasedProcessTree (ProcfsBasedProcessTree.java:isAvailable(181)) - ProcfsBasedProcessTree currently is supported only on Linux.
2015-02-12 16:12:49,796 INFO  mapred.Task (Task.java:initialize(587)) -  Using ResourceCalculatorProcessTree : null
2015-02-12 16:12:49,797 INFO  mapred.ReduceTask (ReduceTask.java:run(362)) - Using ShuffleConsumerPlugin: org.apache.hadoop.mapreduce.task.reduce.Shuffle@50e132fe
2015-02-12 16:12:49,797 INFO  reduce.MergeManagerImpl (MergeManagerImpl.java:<init>(195)) - MergerManager: memoryLimit=493250144, maxSingleShuffleLimit=123312536, mergeThreshold=325545120, ioSortFactor=10, memToMemMergeOutputsThreshold=10
2015-02-12 16:12:49,798 INFO  reduce.EventFetcher (EventFetcher.java:run(61)) - attempt_local1346937241_0015_r_000000_0 Thread started: EventFetcher for fetching Map Completion Events
2015-02-12 16:12:49,798 INFO  reduce.LocalFetcher (LocalFetcher.java:copyMapOutput(141)) - localfetcher#13 about to shuffle output of map attempt_local1346937241_0015_m_000000_0 decomp: 119 len: 123 to MEMORY
2015-02-12 16:12:49,799 INFO  reduce.InMemoryMapOutput (InMemoryMapOutput.java:shuffle(100)) - Read 119 bytes from map-output for attempt_local1346937241_0015_m_000000_0
2015-02-12 16:12:49,799 INFO  reduce.MergeManagerImpl (MergeManagerImpl.java:closeInMemoryFile(313)) - closeInMemoryFile -> map-output of size: 119, inMemoryMapOutputs.size() -> 1, commitMemory -> 0, usedMemory ->119
2015-02-12 16:12:49,799 INFO  reduce.EventFetcher (EventFetcher.java:run(76)) - EventFetcher is interrupted.. Returning
2015-02-12 16:12:49,800 INFO  mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) - 1 / 1 copied.
2015-02-12 16:12:49,800 INFO  reduce.MergeManagerImpl (MergeManagerImpl.java:finalMerge(673)) - finalMerge called with 1 in-memory map-outputs and 0 on-disk map-outputs
2015-02-12 16:12:49,801 INFO  mapred.Merger (Merger.java:merge(606)) - Merging 1 sorted segments
2015-02-12 16:12:49,801 INFO  mapred.Merger (Merger.java:merge(705)) - Down to the last merge-pass, with 1 segments left of total size: 110 bytes
2015-02-12 16:12:49,802 INFO  reduce.MergeManagerImpl (MergeManagerImpl.java:finalMerge(750)) - Merged 1 segments, 119 bytes to disk to satisfy reduce memory limit
2015-02-12 16:12:49,802 INFO  reduce.MergeManagerImpl (MergeManagerImpl.java:finalMerge(780)) - Merging 1 files, 123 bytes from disk
2015-02-12 16:12:49,803 INFO  reduce.MergeManagerImpl (MergeManagerImpl.java:finalMerge(795)) - Merging 0 segments, 0 bytes from memory into reduce
2015-02-12 16:12:49,803 INFO  mapred.Merger (Merger.java:merge(606)) - Merging 1 sorted segments
2015-02-12 16:12:49,803 INFO  mapred.Merger (Merger.java:merge(705)) - Down to the last merge-pass, with 1 segments left of total size: 110 bytes
2015-02-12 16:12:49,803 INFO  mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) - 1 / 1 copied.
2015-02-12 16:12:49,804 INFO  hadoop.FlowReducer (FlowReducer.java:configure(78)) - cascading version: 2.6.3
2015-02-12 16:12:49,804 INFO  hadoop.FlowReducer (FlowReducer.java:configure(79)) - child jvm opts: -Xmx200m
2015-02-12 16:12:49,809 INFO  hadoop.FlowReducer (FlowReducer.java:configure(97)) - sourcing from: GroupBy(sum)[by:[{1}:'char']]
2015-02-12 16:12:49,810 INFO  hadoop.FlowReducer (FlowReducer.java:configure(100)) - sinking to: Hfs["TextDelimited[['char', 'sum']]"]["/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/test/output/hadoop2-mr1/assemblyhelpersplatform/sum"]
2015-02-12 16:12:49,818 INFO  mapred.Task (Task.java:done(1001)) - Task:attempt_local1346937241_0015_r_000000_0 is done. And is in the process of committing
2015-02-12 16:12:49,819 INFO  mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) - 1 / 1 copied.
2015-02-12 16:12:49,819 INFO  mapred.Task (Task.java:commit(1162)) - Task attempt_local1346937241_0015_r_000000_0 is allowed to commit now
2015-02-12 16:12:49,820 INFO  output.FileOutputCommitter (FileOutputCommitter.java:commitTask(439)) - Saved output of task 'attempt_local1346937241_0015_r_000000_0' to file:/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/test/output/hadoop2-mr1/assemblyhelpersplatform/sum/_temporary/0/task_local1346937241_0015_r_000000
2015-02-12 16:12:49,820 INFO  mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) - reduce > reduce
2015-02-12 16:12:49,821 INFO  mapred.Task (Task.java:sendDone(1121)) - Task 'attempt_local1346937241_0015_r_000000_0' done.
2015-02-12 16:12:49,821 INFO  mapred.LocalJobRunner (LocalJobRunner.java:run(325)) - Finishing task: attempt_local1346937241_0015_r_000000_0
2015-02-12 16:12:49,821 INFO  mapred.LocalJobRunner (LocalJobRunner.java:runTasks(456)) - reduce task executor complete.
2015-02-12 16:12:49,841 INFO  util.Hadoop18TapUtil (Hadoop18TapUtil.java:cleanTempPath(219)) - deleting temp path /Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/test/output/hadoop2-mr1/assemblyhelpersplatform/sum/_temporary
2015-02-12 16:12:49,861 INFO  mapred.FileInputFormat (FileInputFormat.java:listStatus(247)) - Total input paths to process : 1
2015-02-12 16:12:49,883 INFO  mapred.FileInputFormat (FileInputFormat.java:listStatus(247)) - Total input paths to process : 1
2015-02-12 16:12:49,884 INFO  cascading.PlatformTestCase (PlatformTestCase.java:tearDown(209)) - copying to local /Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/test/output//hadoop2-mr1/assemblyhelpersplatform/sum
2015-02-12 16:12:49,897 INFO  util.HadoopUtil (HadoopUtil.java:findMainClass(336)) - using default application jar, may cause class not found exceptions on the cluster
2015-02-12 16:12:49,897 INFO  planner.HadoopPlanner (HadoopPlanner.java:initialize(225)) - using application jar: /Users/soberoi/.gradle/caches/modules-2/files-2.1/cascading/cascading-hadoop2-mr1/2.6.3/9ae016e2b8d246b9c1de9168e715d9f1b971aa4a/cascading-hadoop2-mr1-2.6.3.jar
2015-02-12 16:12:49,903 INFO  Configuration.deprecation (Configuration.java:warnOnceIfDeprecated(1022)) - mapred.used.genericoptionsparser is deprecated. Instead, use mapreduce.client.genericoptionsparser.used
2015-02-12 16:12:49,905 INFO  Configuration.deprecation (Configuration.java:warnOnceIfDeprecated(1022)) - mapred.job.tracker is deprecated. Instead, use mapreduce.jobtracker.address
2015-02-12 16:12:49,905 INFO  Configuration.deprecation (Configuration.java:warnOnceIfDeprecated(1022)) - mapred.output.compress is deprecated. Instead, use mapreduce.output.fileoutputformat.compress
2015-02-12 16:12:49,910 INFO  flow.Flow (BaseFlow.java:logInfo(1378)) - [sum-lhs+sum-rhs] starting
2015-02-12 16:12:49,910 INFO  flow.Flow (BaseFlow.java:logInfo(1378)) - [sum-lhs+sum-rhs]  source: Hfs["TextDelimited[['num', 'char']]"]["/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/classes/test/data/rhs.txt"]
2015-02-12 16:12:49,911 INFO  flow.Flow (BaseFlow.java:logInfo(1378)) - [sum-lhs+sum-rhs]  source: Hfs["TextDelimited[['num', 'char']]"]["/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/classes/test/data/lhs.txt"]
2015-02-12 16:12:49,911 INFO  flow.Flow (BaseFlow.java:logInfo(1378)) - [sum-lhs+sum-rhs]  sink: Hfs["TextDelimited[['char', 'sum']]"]["/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/test/output/hadoop2-mr1/assemblyhelpersplatform/mergesum"]
2015-02-12 16:12:49,911 INFO  flow.Flow (BaseFlow.java:logInfo(1378)) - [sum-lhs+sum-rhs]  parallel execution is enabled: false
2015-02-12 16:12:49,911 INFO  flow.Flow (BaseFlow.java:logInfo(1378)) - [sum-lhs+sum-rhs]  starting jobs: 1
2015-02-12 16:12:49,911 INFO  flow.Flow (BaseFlow.java:logInfo(1378)) - [sum-lhs+sum-rhs]  allocating threads: 1
2015-02-12 16:12:49,911 INFO  flow.FlowStep (BaseFlowStep.java:logInfo(834)) - [sum-lhs+sum-rhs] starting step: (1/1) ...yhelpersplatform/mergesum
2015-02-12 16:12:49,912 INFO  jvm.JvmMetrics (JvmMetrics.java:init(71)) - Cannot initialize JVM Metrics with processName=JobTracker, sessionId= - already initialized
2015-02-12 16:12:49,913 INFO  jvm.JvmMetrics (JvmMetrics.java:init(71)) - Cannot initialize JVM Metrics with processName=JobTracker, sessionId= - already initialized
2015-02-12 16:12:49,931 INFO  mapred.FileInputFormat (FileInputFormat.java:listStatus(247)) - Total input paths to process : 1
2015-02-12 16:12:49,932 INFO  mapred.FileInputFormat (FileInputFormat.java:listStatus(247)) - Total input paths to process : 1
2015-02-12 16:12:49,953 INFO  mapreduce.JobSubmitter (JobSubmitter.java:submitJobInternal(492)) - number of splits:2
2015-02-12 16:12:49,964 INFO  mapreduce.JobSubmitter (JobSubmitter.java:printTokens(575)) - Submitting tokens for job: job_local1316284914_0016
2015-02-12 16:12:49,976 WARN  conf.Configuration (Configuration.java:loadProperty(2510)) - file:/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/tmp/cascading/staging/soberoi1316284914/.staging/job_local1316284914_0016/job.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.retry.interval;  Ignoring.
2015-02-12 16:12:49,977 WARN  conf.Configuration (Configuration.java:loadProperty(2510)) - file:/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/tmp/cascading/staging/soberoi1316284914/.staging/job_local1316284914_0016/job.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.attempts;  Ignoring.
2015-02-12 16:12:50,008 WARN  conf.Configuration (Configuration.java:loadProperty(2510)) - file:/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/test/tmp/mapred/local/localRunner/soberoi/job_local1316284914_0016/job_local1316284914_0016.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.retry.interval;  Ignoring.
2015-02-12 16:12:50,009 WARN  conf.Configuration (Configuration.java:loadProperty(2510)) - file:/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/test/tmp/mapred/local/localRunner/soberoi/job_local1316284914_0016/job_local1316284914_0016.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.attempts;  Ignoring.
2015-02-12 16:12:50,009 INFO  mapreduce.Job (Job.java:submit(1299)) - The url to track the job: http://localhost:8080/
2015-02-12 16:12:50,009 INFO  mapred.LocalJobRunner (LocalJobRunner.java:createOutputCommitter(471)) - OutputCommitter set in config null
2015-02-12 16:12:50,010 INFO  mapred.LocalJobRunner (LocalJobRunner.java:createOutputCommitter(489)) - OutputCommitter is org.apache.hadoop.mapred.FileOutputCommitter
2015-02-12 16:12:50,009 INFO  flow.FlowStep (BaseFlowStep.java:logInfo(834)) - [sum-lhs+sum-rhs] submitted hadoop job: job_local1316284914_0016
2015-02-12 16:12:50,010 INFO  flow.FlowStep (BaseFlowStep.java:logInfo(834)) - [sum-lhs+sum-rhs] tracking url: http://localhost:8080/
2015-02-12 16:12:50,011 INFO  mapred.LocalJobRunner (LocalJobRunner.java:runTasks(448)) - Waiting for map tasks
2015-02-12 16:12:50,012 INFO  mapred.LocalJobRunner (LocalJobRunner.java:run(224)) - Starting task: attempt_local1316284914_0016_m_000000_0
2015-02-12 16:12:50,013 INFO  util.ProcfsBasedProcessTree (ProcfsBasedProcessTree.java:isAvailable(181)) - ProcfsBasedProcessTree currently is supported only on Linux.
2015-02-12 16:12:50,013 INFO  mapred.Task (Task.java:initialize(587)) -  Using ResourceCalculatorProcessTree : null
2015-02-12 16:12:50,013 INFO  io.MultiInputSplit (MultiInputSplit.java:readFields(161)) - current split input path: file:/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/classes/test/data/lhs.txt
2015-02-12 16:12:50,014 INFO  mapred.MapTask (MapTask.java:updateJobWithSplit(482)) - Processing split: cascading.tap.hadoop.io.MultiInputSplit@3ae24197
2015-02-12 16:12:50,014 INFO  mapred.MapTask (MapTask.java:runOldMapper(436)) - numReduceTasks: 1
2015-02-12 16:12:50,029 INFO  mapred.MapTask (MapTask.java:setEquator(1202)) - (EQUATOR) 0 kvi 26214396(104857584)
2015-02-12 16:12:50,030 INFO  mapred.MapTask (MapTask.java:init(995)) - mapreduce.task.io.sort.mb: 100
2015-02-12 16:12:50,030 INFO  mapred.MapTask (MapTask.java:init(996)) - soft limit at 83886080
2015-02-12 16:12:50,030 INFO  mapred.MapTask (MapTask.java:init(997)) - bufstart = 0; bufvoid = 104857600
2015-02-12 16:12:50,030 INFO  mapred.MapTask (MapTask.java:init(998)) - kvstart = 26214396; length = 6553600
2015-02-12 16:12:50,031 INFO  mapred.MapTask (MapTask.java:createSortingCollector(402)) - Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer
2015-02-12 16:12:50,031 INFO  hadoop.FlowMapper (FlowMapper.java:configure(71)) - cascading version: 2.6.3
2015-02-12 16:12:50,031 INFO  hadoop.FlowMapper (FlowMapper.java:configure(72)) - child jvm opts: -Xmx200m
2015-02-12 16:12:50,038 INFO  hadoop.FlowMapper (FlowMapper.java:configure(87)) - sourcing from: Hfs["TextDelimited[['num', 'char']]"]["/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/classes/test/data/lhs.txt"]
2015-02-12 16:12:50,038 INFO  hadoop.FlowMapper (FlowMapper.java:configure(90)) - sinking to: GroupBy(sum-lhs+sum-rhs)[by: sum-lhs:[{1}:'char'] sum-rhs:[{1}:'char']]
2015-02-12 16:12:50,038 INFO  assembly.AggregateBy (AggregateBy.java:prepare(267)) - using threshold value: 2
2015-02-12 16:12:50,039 INFO  assembly.AggregateBy (AggregateBy.java:removeEldestEntry(327)) - flushed keys num times: 1, with threshold: 2
2015-02-12 16:12:50,039 INFO  assembly.AggregateBy (AggregateBy.java:removeEldestEntry(328)) - mem on flush (mb), free: 197, total: 509, max: 672
2015-02-12 16:12:50,039 INFO  assembly.AggregateBy (AggregateBy.java:removeEldestEntry(333)) - total mem is 75% of max mem, to better utilize unused memory consider increasing current LRU threshold with system property "cascading.aggregateby.threshold"
2015-02-12 16:12:50,040 INFO  assembly.AggregateBy (AggregateBy.java:removeEldestEntry(327)) - flushed keys num times: 3, with threshold: 2
2015-02-12 16:12:50,040 INFO  assembly.AggregateBy (AggregateBy.java:removeEldestEntry(328)) - mem on flush (mb), free: 197, total: 509, max: 672
2015-02-12 16:12:50,040 INFO  assembly.AggregateBy (AggregateBy.java:removeEldestEntry(333)) - total mem is 75% of max mem, to better utilize unused memory consider increasing current LRU threshold with system property "cascading.aggregateby.threshold"
2015-02-12 16:12:50,040 INFO  assembly.AggregateBy (AggregateBy.java:removeEldestEntry(327)) - flushed keys num times: 5, with threshold: 2
2015-02-12 16:12:50,040 INFO  assembly.AggregateBy (AggregateBy.java:removeEldestEntry(328)) - mem on flush (mb), free: 197, total: 509, max: 672
2015-02-12 16:12:50,041 INFO  assembly.AggregateBy (AggregateBy.java:removeEldestEntry(333)) - total mem is 75% of max mem, to better utilize unused memory consider increasing current LRU threshold with system property "cascading.aggregateby.threshold"
2015-02-12 16:12:50,041 INFO  assembly.AggregateBy (AggregateBy.java:removeEldestEntry(327)) - flushed keys num times: 7, with threshold: 2
2015-02-12 16:12:50,041 INFO  assembly.AggregateBy (AggregateBy.java:removeEldestEntry(328)) - mem on flush (mb), free: 197, total: 509, max: 672
2015-02-12 16:12:50,041 INFO  assembly.AggregateBy (AggregateBy.java:removeEldestEntry(333)) - total mem is 75% of max mem, to better utilize unused memory consider increasing current LRU threshold with system property "cascading.aggregateby.threshold"
2015-02-12 16:12:50,042 INFO  mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) - 
2015-02-12 16:12:50,042 INFO  mapred.MapTask (MapTask.java:flush(1457)) - Starting flush of map output
2015-02-12 16:12:50,042 INFO  mapred.MapTask (MapTask.java:flush(1475)) - Spilling map output
2015-02-12 16:12:50,042 INFO  mapred.MapTask (MapTask.java:flush(1476)) - bufstart = 0; bufend = 99; bufvoid = 104857600
2015-02-12 16:12:50,042 INFO  mapred.MapTask (MapTask.java:flush(1478)) - kvstart = 26214396(104857584); kvend = 26214364(104857456); length = 33/6553600
2015-02-12 16:12:50,044 INFO  mapred.MapTask (MapTask.java:sortAndSpill(1660)) - Finished spill 0
2015-02-12 16:12:50,045 INFO  mapred.Task (Task.java:done(1001)) - Task:attempt_local1316284914_0016_m_000000_0 is done. And is in the process of committing
2015-02-12 16:12:50,047 INFO  mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) - file:/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/classes/test/data/lhs.txt:0+52
2015-02-12 16:12:50,047 INFO  mapred.Task (Task.java:sendDone(1121)) - Task 'attempt_local1316284914_0016_m_000000_0' done.
2015-02-12 16:12:50,047 INFO  mapred.LocalJobRunner (LocalJobRunner.java:run(249)) - Finishing task: attempt_local1316284914_0016_m_000000_0
2015-02-12 16:12:50,047 INFO  mapred.LocalJobRunner (LocalJobRunner.java:run(224)) - Starting task: attempt_local1316284914_0016_m_000001_0
2015-02-12 16:12:50,048 INFO  util.ProcfsBasedProcessTree (ProcfsBasedProcessTree.java:isAvailable(181)) - ProcfsBasedProcessTree currently is supported only on Linux.
2015-02-12 16:12:50,048 INFO  mapred.Task (Task.java:initialize(587)) -  Using ResourceCalculatorProcessTree : null
2015-02-12 16:12:50,050 INFO  io.MultiInputSplit (MultiInputSplit.java:readFields(161)) - current split input path: file:/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/classes/test/data/rhs.txt
2015-02-12 16:12:50,050 INFO  mapred.MapTask (MapTask.java:updateJobWithSplit(482)) - Processing split: cascading.tap.hadoop.io.MultiInputSplit@74a32d61
2015-02-12 16:12:50,051 INFO  mapred.MapTask (MapTask.java:runOldMapper(436)) - numReduceTasks: 1
2015-02-12 16:12:50,063 INFO  mapred.MapTask (MapTask.java:setEquator(1202)) - (EQUATOR) 0 kvi 26214396(104857584)
2015-02-12 16:12:50,063 INFO  mapred.MapTask (MapTask.java:init(995)) - mapreduce.task.io.sort.mb: 100
2015-02-12 16:12:50,063 INFO  mapred.MapTask (MapTask.java:init(996)) - soft limit at 83886080
2015-02-12 16:12:50,064 INFO  mapred.MapTask (MapTask.java:init(997)) - bufstart = 0; bufvoid = 104857600
2015-02-12 16:12:50,064 INFO  mapred.MapTask (MapTask.java:init(998)) - kvstart = 26214396; length = 6553600
2015-02-12 16:12:50,064 INFO  mapred.MapTask (MapTask.java:createSortingCollector(402)) - Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer
2015-02-12 16:12:50,065 INFO  hadoop.FlowMapper (FlowMapper.java:configure(71)) - cascading version: 2.6.3
2015-02-12 16:12:50,065 INFO  hadoop.FlowMapper (FlowMapper.java:configure(72)) - child jvm opts: -Xmx200m
2015-02-12 16:12:50,070 INFO  hadoop.FlowMapper (FlowMapper.java:configure(87)) - sourcing from: Hfs["TextDelimited[['num', 'char']]"]["/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/classes/test/data/rhs.txt"]
2015-02-12 16:12:50,070 INFO  hadoop.FlowMapper (FlowMapper.java:configure(90)) - sinking to: GroupBy(sum-lhs+sum-rhs)[by: sum-lhs:[{1}:'char'] sum-rhs:[{1}:'char']]
2015-02-12 16:12:50,071 INFO  assembly.AggregateBy (AggregateBy.java:prepare(267)) - using threshold value: 2
2015-02-12 16:12:50,073 INFO  assembly.AggregateBy (AggregateBy.java:removeEldestEntry(327)) - flushed keys num times: 1, with threshold: 2
2015-02-12 16:12:50,073 INFO  assembly.AggregateBy (AggregateBy.java:removeEldestEntry(328)) - mem on flush (mb), free: 95, total: 509, max: 672
2015-02-12 16:12:50,073 INFO  assembly.AggregateBy (AggregateBy.java:removeEldestEntry(333)) - total mem is 75% of max mem, to better utilize unused memory consider increasing current LRU threshold with system property "cascading.aggregateby.threshold"
2015-02-12 16:12:50,074 INFO  assembly.AggregateBy (AggregateBy.java:removeEldestEntry(327)) - flushed keys num times: 3, with threshold: 2
2015-02-12 16:12:50,074 INFO  assembly.AggregateBy (AggregateBy.java:removeEldestEntry(328)) - mem on flush (mb), free: 95, total: 509, max: 672
2015-02-12 16:12:50,074 INFO  assembly.AggregateBy (AggregateBy.java:removeEldestEntry(333)) - total mem is 75% of max mem, to better utilize unused memory consider increasing current LRU threshold with system property "cascading.aggregateby.threshold"
2015-02-12 16:12:50,075 INFO  assembly.AggregateBy (AggregateBy.java:removeEldestEntry(327)) - flushed keys num times: 5, with threshold: 2
2015-02-12 16:12:50,075 INFO  assembly.AggregateBy (AggregateBy.java:removeEldestEntry(328)) - mem on flush (mb), free: 95, total: 509, max: 672
2015-02-12 16:12:50,075 INFO  assembly.AggregateBy (AggregateBy.java:removeEldestEntry(333)) - total mem is 75% of max mem, to better utilize unused memory consider increasing current LRU threshold with system property "cascading.aggregateby.threshold"
2015-02-12 16:12:50,075 INFO  assembly.AggregateBy (AggregateBy.java:removeEldestEntry(327)) - flushed keys num times: 7, with threshold: 2
2015-02-12 16:12:50,075 INFO  assembly.AggregateBy (AggregateBy.java:removeEldestEntry(328)) - mem on flush (mb), free: 95, total: 509, max: 672
2015-02-12 16:12:50,076 INFO  assembly.AggregateBy (AggregateBy.java:removeEldestEntry(333)) - total mem is 75% of max mem, to better utilize unused memory consider increasing current LRU threshold with system property "cascading.aggregateby.threshold"
2015-02-12 16:12:50,076 INFO  mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) - 
2015-02-12 16:12:50,076 INFO  mapred.MapTask (MapTask.java:flush(1457)) - Starting flush of map output
2015-02-12 16:12:50,076 INFO  mapred.MapTask (MapTask.java:flush(1475)) - Spilling map output
2015-02-12 16:12:50,077 INFO  mapred.MapTask (MapTask.java:flush(1476)) - bufstart = 0; bufend = 99; bufvoid = 104857600
2015-02-12 16:12:50,077 INFO  mapred.MapTask (MapTask.java:flush(1478)) - kvstart = 26214396(104857584); kvend = 26214364(104857456); length = 33/6553600
2015-02-12 16:12:50,078 INFO  mapred.MapTask (MapTask.java:sortAndSpill(1660)) - Finished spill 0
2015-02-12 16:12:50,080 INFO  mapred.Task (Task.java:done(1001)) - Task:attempt_local1316284914_0016_m_000001_0 is done. And is in the process of committing
2015-02-12 16:12:50,081 INFO  mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) - file:/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/classes/test/data/rhs.txt:0+51
2015-02-12 16:12:50,081 INFO  mapred.Task (Task.java:sendDone(1121)) - Task 'attempt_local1316284914_0016_m_000001_0' done.
2015-02-12 16:12:50,081 INFO  mapred.LocalJobRunner (LocalJobRunner.java:run(249)) - Finishing task: attempt_local1316284914_0016_m_000001_0
2015-02-12 16:12:50,081 INFO  mapred.LocalJobRunner (LocalJobRunner.java:runTasks(456)) - map task executor complete.
2015-02-12 16:12:50,082 INFO  mapred.LocalJobRunner (LocalJobRunner.java:runTasks(448)) - Waiting for reduce tasks
2015-02-12 16:12:50,082 INFO  mapred.LocalJobRunner (LocalJobRunner.java:run(302)) - Starting task: attempt_local1316284914_0016_r_000000_0
2015-02-12 16:12:50,083 INFO  util.ProcfsBasedProcessTree (ProcfsBasedProcessTree.java:isAvailable(181)) - ProcfsBasedProcessTree currently is supported only on Linux.
2015-02-12 16:12:50,083 INFO  mapred.Task (Task.java:initialize(587)) -  Using ResourceCalculatorProcessTree : null
2015-02-12 16:12:50,083 INFO  mapred.ReduceTask (ReduceTask.java:run(362)) - Using ShuffleConsumerPlugin: org.apache.hadoop.mapreduce.task.reduce.Shuffle@25719f5c
2015-02-12 16:12:50,083 INFO  reduce.MergeManagerImpl (MergeManagerImpl.java:<init>(195)) - MergerManager: memoryLimit=493250144, maxSingleShuffleLimit=123312536, mergeThreshold=325545120, ioSortFactor=10, memToMemMergeOutputsThreshold=10
2015-02-12 16:12:50,084 INFO  reduce.EventFetcher (EventFetcher.java:run(61)) - attempt_local1316284914_0016_r_000000_0 Thread started: EventFetcher for fetching Map Completion Events
2015-02-12 16:12:50,084 INFO  reduce.LocalFetcher (LocalFetcher.java:copyMapOutput(141)) - localfetcher#14 about to shuffle output of map attempt_local1316284914_0016_m_000001_0 decomp: 119 len: 123 to MEMORY
2015-02-12 16:12:50,085 INFO  reduce.InMemoryMapOutput (InMemoryMapOutput.java:shuffle(100)) - Read 119 bytes from map-output for attempt_local1316284914_0016_m_000001_0
2015-02-12 16:12:50,085 INFO  reduce.MergeManagerImpl (MergeManagerImpl.java:closeInMemoryFile(313)) - closeInMemoryFile -> map-output of size: 119, inMemoryMapOutputs.size() -> 1, commitMemory -> 0, usedMemory ->119
2015-02-12 16:12:50,086 INFO  reduce.LocalFetcher (LocalFetcher.java:copyMapOutput(141)) - localfetcher#14 about to shuffle output of map attempt_local1316284914_0016_m_000000_0 decomp: 119 len: 123 to MEMORY
2015-02-12 16:12:50,086 INFO  reduce.InMemoryMapOutput (InMemoryMapOutput.java:shuffle(100)) - Read 119 bytes from map-output for attempt_local1316284914_0016_m_000000_0
2015-02-12 16:12:50,086 INFO  reduce.MergeManagerImpl (MergeManagerImpl.java:closeInMemoryFile(313)) - closeInMemoryFile -> map-output of size: 119, inMemoryMapOutputs.size() -> 2, commitMemory -> 119, usedMemory ->238
2015-02-12 16:12:50,087 INFO  reduce.EventFetcher (EventFetcher.java:run(76)) - EventFetcher is interrupted.. Returning
2015-02-12 16:12:50,087 INFO  mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) - 2 / 2 copied.
2015-02-12 16:12:50,087 INFO  reduce.MergeManagerImpl (MergeManagerImpl.java:finalMerge(673)) - finalMerge called with 2 in-memory map-outputs and 0 on-disk map-outputs
2015-02-12 16:12:50,089 INFO  mapred.Merger (Merger.java:merge(606)) - Merging 2 sorted segments
2015-02-12 16:12:50,089 INFO  mapred.Merger (Merger.java:merge(705)) - Down to the last merge-pass, with 2 segments left of total size: 220 bytes
2015-02-12 16:12:50,090 INFO  reduce.MergeManagerImpl (MergeManagerImpl.java:finalMerge(750)) - Merged 2 segments, 238 bytes to disk to satisfy reduce memory limit
2015-02-12 16:12:50,091 INFO  reduce.MergeManagerImpl (MergeManagerImpl.java:finalMerge(780)) - Merging 1 files, 240 bytes from disk
2015-02-12 16:12:50,091 INFO  reduce.MergeManagerImpl (MergeManagerImpl.java:finalMerge(795)) - Merging 0 segments, 0 bytes from memory into reduce
2015-02-12 16:12:50,091 INFO  mapred.Merger (Merger.java:merge(606)) - Merging 1 sorted segments
2015-02-12 16:12:50,091 INFO  mapred.Merger (Merger.java:merge(705)) - Down to the last merge-pass, with 1 segments left of total size: 227 bytes
2015-02-12 16:12:50,092 INFO  mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) - 2 / 2 copied.
2015-02-12 16:12:50,092 INFO  hadoop.FlowReducer (FlowReducer.java:configure(78)) - cascading version: 2.6.3
2015-02-12 16:12:50,092 INFO  hadoop.FlowReducer (FlowReducer.java:configure(79)) - child jvm opts: -Xmx200m
2015-02-12 16:12:50,097 INFO  hadoop.FlowReducer (FlowReducer.java:configure(97)) - sourcing from: GroupBy(sum-lhs+sum-rhs)[by: sum-lhs:[{1}:'char'] sum-rhs:[{1}:'char']]
2015-02-12 16:12:50,097 INFO  hadoop.FlowReducer (FlowReducer.java:configure(100)) - sinking to: Hfs["TextDelimited[['char', 'sum']]"]["/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/test/output/hadoop2-mr1/assemblyhelpersplatform/mergesum"]
2015-02-12 16:12:50,106 INFO  mapred.Task (Task.java:done(1001)) - Task:attempt_local1316284914_0016_r_000000_0 is done. And is in the process of committing
2015-02-12 16:12:50,107 INFO  mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) - 2 / 2 copied.
2015-02-12 16:12:50,107 INFO  mapred.Task (Task.java:commit(1162)) - Task attempt_local1316284914_0016_r_000000_0 is allowed to commit now
2015-02-12 16:12:50,108 INFO  output.FileOutputCommitter (FileOutputCommitter.java:commitTask(439)) - Saved output of task 'attempt_local1316284914_0016_r_000000_0' to file:/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/test/output/hadoop2-mr1/assemblyhelpersplatform/mergesum/_temporary/0/task_local1316284914_0016_r_000000
2015-02-12 16:12:50,109 INFO  mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) - reduce > reduce
2015-02-12 16:12:50,109 INFO  mapred.Task (Task.java:sendDone(1121)) - Task 'attempt_local1316284914_0016_r_000000_0' done.
2015-02-12 16:12:50,109 INFO  mapred.LocalJobRunner (LocalJobRunner.java:run(325)) - Finishing task: attempt_local1316284914_0016_r_000000_0
2015-02-12 16:12:50,109 INFO  mapred.LocalJobRunner (LocalJobRunner.java:runTasks(456)) - reduce task executor complete.
2015-02-12 16:12:50,124 INFO  util.Hadoop18TapUtil (Hadoop18TapUtil.java:cleanTempPath(219)) - deleting temp path /Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/test/output/hadoop2-mr1/assemblyhelpersplatform/mergesum/_temporary
2015-02-12 16:12:50,148 INFO  mapred.FileInputFormat (FileInputFormat.java:listStatus(247)) - Total input paths to process : 1
2015-02-12 16:12:50,171 INFO  mapred.FileInputFormat (FileInputFormat.java:listStatus(247)) - Total input paths to process : 1
2015-02-12 16:12:50,172 INFO  cascading.PlatformTestCase (PlatformTestCase.java:tearDown(209)) - copying to local /Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/test/output//hadoop2-mr1/assemblyhelpersplatform/mergesum
2015-02-12 16:12:50,191 INFO  util.HadoopUtil (HadoopUtil.java:findMainClass(336)) - using default application jar, may cause class not found exceptions on the cluster
2015-02-12 16:12:50,192 INFO  planner.HadoopPlanner (HadoopPlanner.java:initialize(225)) - using application jar: /Users/soberoi/.gradle/caches/modules-2/files-2.1/cascading/cascading-hadoop2-mr1/2.6.3/9ae016e2b8d246b9c1de9168e715d9f1b971aa4a/cascading-hadoop2-mr1-2.6.3.jar
2015-02-12 16:12:50,196 INFO  Configuration.deprecation (Configuration.java:warnOnceIfDeprecated(1022)) - mapred.used.genericoptionsparser is deprecated. Instead, use mapreduce.client.genericoptionsparser.used
2015-02-12 16:12:50,197 INFO  Configuration.deprecation (Configuration.java:warnOnceIfDeprecated(1022)) - mapred.job.tracker is deprecated. Instead, use mapreduce.jobtracker.address
2015-02-12 16:12:50,197 INFO  Configuration.deprecation (Configuration.java:warnOnceIfDeprecated(1022)) - mapred.output.compress is deprecated. Instead, use mapreduce.output.fileoutputformat.compress
2015-02-12 16:12:50,202 INFO  flow.Flow (BaseFlow.java:logInfo(1378)) - [shape] starting
2015-02-12 16:12:50,202 INFO  flow.Flow (BaseFlow.java:logInfo(1378)) - [shape]  source: Hfs["TextLine[['offset', 'line']->[ALL]]"]["/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/classes/test/data/lower.txt"]
2015-02-12 16:12:50,202 INFO  flow.Flow (BaseFlow.java:logInfo(1378)) - [shape]  sink: Hfs["TextLine[['num']]"]["/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/test/output/hadoop2-mr1/assemblyhelpersplatform/discardnarrow"]
2015-02-12 16:12:50,202 INFO  flow.Flow (BaseFlow.java:logInfo(1378)) - [shape]  parallel execution is enabled: false
2015-02-12 16:12:50,202 INFO  flow.Flow (BaseFlow.java:logInfo(1378)) - [shape]  starting jobs: 1
2015-02-12 16:12:50,202 INFO  flow.Flow (BaseFlow.java:logInfo(1378)) - [shape]  allocating threads: 1
2015-02-12 16:12:50,203 INFO  flow.FlowStep (BaseFlowStep.java:logInfo(834)) - [shape] starting step: (1/1) ...ersplatform/discardnarrow
2015-02-12 16:12:50,203 INFO  jvm.JvmMetrics (JvmMetrics.java:init(71)) - Cannot initialize JVM Metrics with processName=JobTracker, sessionId= - already initialized
2015-02-12 16:12:50,204 INFO  jvm.JvmMetrics (JvmMetrics.java:init(71)) - Cannot initialize JVM Metrics with processName=JobTracker, sessionId= - already initialized
2015-02-12 16:12:50,222 INFO  mapred.FileInputFormat (FileInputFormat.java:listStatus(247)) - Total input paths to process : 1
2015-02-12 16:12:50,238 INFO  mapreduce.JobSubmitter (JobSubmitter.java:submitJobInternal(492)) - number of splits:1
2015-02-12 16:12:50,249 INFO  mapreduce.JobSubmitter (JobSubmitter.java:printTokens(575)) - Submitting tokens for job: job_local1214593860_0017
2015-02-12 16:12:50,261 WARN  conf.Configuration (Configuration.java:loadProperty(2510)) - file:/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/tmp/cascading/staging/soberoi1214593860/.staging/job_local1214593860_0017/job.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.retry.interval;  Ignoring.
2015-02-12 16:12:50,262 WARN  conf.Configuration (Configuration.java:loadProperty(2510)) - file:/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/tmp/cascading/staging/soberoi1214593860/.staging/job_local1214593860_0017/job.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.attempts;  Ignoring.
2015-02-12 16:12:50,293 WARN  conf.Configuration (Configuration.java:loadProperty(2510)) - file:/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/test/tmp/mapred/local/localRunner/soberoi/job_local1214593860_0017/job_local1214593860_0017.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.retry.interval;  Ignoring.
2015-02-12 16:12:50,294 WARN  conf.Configuration (Configuration.java:loadProperty(2510)) - file:/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/test/tmp/mapred/local/localRunner/soberoi/job_local1214593860_0017/job_local1214593860_0017.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.attempts;  Ignoring.
2015-02-12 16:12:50,294 INFO  mapreduce.Job (Job.java:submit(1299)) - The url to track the job: http://localhost:8080/
2015-02-12 16:12:50,294 INFO  mapred.LocalJobRunner (LocalJobRunner.java:createOutputCommitter(471)) - OutputCommitter set in config null
2015-02-12 16:12:50,294 INFO  flow.FlowStep (BaseFlowStep.java:logInfo(834)) - [shape] submitted hadoop job: job_local1214593860_0017
2015-02-12 16:12:50,295 INFO  flow.FlowStep (BaseFlowStep.java:logInfo(834)) - [shape] tracking url: http://localhost:8080/
2015-02-12 16:12:50,295 INFO  mapred.LocalJobRunner (LocalJobRunner.java:createOutputCommitter(489)) - OutputCommitter is org.apache.hadoop.mapred.FileOutputCommitter
2015-02-12 16:12:50,296 INFO  mapred.LocalJobRunner (LocalJobRunner.java:runTasks(448)) - Waiting for map tasks
2015-02-12 16:12:50,296 INFO  mapred.LocalJobRunner (LocalJobRunner.java:run(224)) - Starting task: attempt_local1214593860_0017_m_000000_0
2015-02-12 16:12:50,297 INFO  util.ProcfsBasedProcessTree (ProcfsBasedProcessTree.java:isAvailable(181)) - ProcfsBasedProcessTree currently is supported only on Linux.
2015-02-12 16:12:50,297 INFO  mapred.Task (Task.java:initialize(587)) -  Using ResourceCalculatorProcessTree : null
2015-02-12 16:12:50,298 INFO  io.MultiInputSplit (MultiInputSplit.java:readFields(161)) - current split input path: file:/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/classes/test/data/lower.txt
2015-02-12 16:12:50,298 INFO  mapred.MapTask (MapTask.java:updateJobWithSplit(482)) - Processing split: cascading.tap.hadoop.io.MultiInputSplit@221cf05
2015-02-12 16:12:50,299 INFO  mapred.MapTask (MapTask.java:runOldMapper(436)) - numReduceTasks: 0
2015-02-12 16:12:50,304 INFO  hadoop.FlowMapper (FlowMapper.java:configure(71)) - cascading version: 2.6.3
2015-02-12 16:12:50,304 INFO  hadoop.FlowMapper (FlowMapper.java:configure(72)) - child jvm opts: -Xmx200m
2015-02-12 16:12:50,307 INFO  hadoop.FlowMapper (FlowMapper.java:configure(87)) - sourcing from: Hfs["TextLine[['offset', 'line']->[ALL]]"]["/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/classes/test/data/lower.txt"]
2015-02-12 16:12:50,307 INFO  hadoop.FlowMapper (FlowMapper.java:configure(90)) - sinking to: Hfs["TextLine[['num']]"]["/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/test/output/hadoop2-mr1/assemblyhelpersplatform/discardnarrow"]
2015-02-12 16:12:50,308 INFO  mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) - 
2015-02-12 16:12:50,309 INFO  mapred.Task (Task.java:done(1001)) - Task:attempt_local1214593860_0017_m_000000_0 is done. And is in the process of committing
2015-02-12 16:12:50,310 INFO  mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) - 
2015-02-12 16:12:50,310 INFO  mapred.Task (Task.java:commit(1162)) - Task attempt_local1214593860_0017_m_000000_0 is allowed to commit now
2015-02-12 16:12:50,310 INFO  output.FileOutputCommitter (FileOutputCommitter.java:commitTask(439)) - Saved output of task 'attempt_local1214593860_0017_m_000000_0' to file:/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/test/output/hadoop2-mr1/assemblyhelpersplatform/discardnarrow/_temporary/0/task_local1214593860_0017_m_000000
2015-02-12 16:12:50,311 INFO  mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) - file:/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/classes/test/data/lower.txt:0+19
2015-02-12 16:12:50,311 INFO  mapred.Task (Task.java:sendDone(1121)) - Task 'attempt_local1214593860_0017_m_000000_0' done.
2015-02-12 16:12:50,311 INFO  mapred.LocalJobRunner (LocalJobRunner.java:run(249)) - Finishing task: attempt_local1214593860_0017_m_000000_0
2015-02-12 16:12:50,311 INFO  mapred.LocalJobRunner (LocalJobRunner.java:runTasks(456)) - map task executor complete.
2015-02-12 16:12:50,322 INFO  util.Hadoop18TapUtil (Hadoop18TapUtil.java:cleanTempPath(219)) - deleting temp path /Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/test/output/hadoop2-mr1/assemblyhelpersplatform/discardnarrow/_temporary
2015-02-12 16:12:50,350 INFO  mapred.FileInputFormat (FileInputFormat.java:listStatus(247)) - Total input paths to process : 1
2015-02-12 16:12:50,351 INFO  cascading.PlatformTestCase (PlatformTestCase.java:tearDown(209)) - copying to local /Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/test/output//hadoop2-mr1/assemblyhelpersplatform/discardnarrow
2015-02-12 16:12:50,362 INFO  util.HadoopUtil (HadoopUtil.java:findMainClass(336)) - using default application jar, may cause class not found exceptions on the cluster
2015-02-12 16:12:50,362 INFO  planner.HadoopPlanner (HadoopPlanner.java:initialize(225)) - using application jar: /Users/soberoi/.gradle/caches/modules-2/files-2.1/cascading/cascading-hadoop2-mr1/2.6.3/9ae016e2b8d246b9c1de9168e715d9f1b971aa4a/cascading-hadoop2-mr1-2.6.3.jar
2015-02-12 16:12:50,367 INFO  Configuration.deprecation (Configuration.java:warnOnceIfDeprecated(1022)) - mapred.used.genericoptionsparser is deprecated. Instead, use mapreduce.client.genericoptionsparser.used
2015-02-12 16:12:50,369 INFO  Configuration.deprecation (Configuration.java:warnOnceIfDeprecated(1022)) - mapred.job.tracker is deprecated. Instead, use mapreduce.jobtracker.address
2015-02-12 16:12:50,369 INFO  Configuration.deprecation (Configuration.java:warnOnceIfDeprecated(1022)) - mapred.output.compress is deprecated. Instead, use mapreduce.output.fileoutputformat.compress
2015-02-12 16:12:50,374 INFO  flow.Flow (BaseFlow.java:logInfo(1378)) - [shape] starting
2015-02-12 16:12:50,375 INFO  flow.Flow (BaseFlow.java:logInfo(1378)) - [shape]  source: Hfs["TextLine[['offset', 'line']->[ALL]]"]["/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/classes/test/data/lower.txt"]
2015-02-12 16:12:50,375 INFO  flow.Flow (BaseFlow.java:logInfo(1378)) - [shape]  sink: Hfs["TextLine[['line']->['item', 'element']]"]["/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/test/output/hadoop2-mr1/assemblyhelpersplatform/rename"]
2015-02-12 16:12:50,375 INFO  flow.Flow (BaseFlow.java:logInfo(1378)) - [shape]  parallel execution is enabled: false
2015-02-12 16:12:50,375 INFO  flow.Flow (BaseFlow.java:logInfo(1378)) - [shape]  starting jobs: 1
2015-02-12 16:12:50,375 INFO  flow.Flow (BaseFlow.java:logInfo(1378)) - [shape]  allocating threads: 1
2015-02-12 16:12:50,376 INFO  flow.FlowStep (BaseFlowStep.java:logInfo(834)) - [shape] starting step: (1/1) ...blyhelpersplatform/rename
2015-02-12 16:12:50,377 INFO  jvm.JvmMetrics (JvmMetrics.java:init(71)) - Cannot initialize JVM Metrics with processName=JobTracker, sessionId= - already initialized
2015-02-12 16:12:50,378 INFO  jvm.JvmMetrics (JvmMetrics.java:init(71)) - Cannot initialize JVM Metrics with processName=JobTracker, sessionId= - already initialized
2015-02-12 16:12:50,400 INFO  mapred.FileInputFormat (FileInputFormat.java:listStatus(247)) - Total input paths to process : 1
2015-02-12 16:12:50,416 INFO  mapreduce.JobSubmitter (JobSubmitter.java:submitJobInternal(492)) - number of splits:1
2015-02-12 16:12:50,427 INFO  mapreduce.JobSubmitter (JobSubmitter.java:printTokens(575)) - Submitting tokens for job: job_local90064939_0018
2015-02-12 16:12:50,441 WARN  conf.Configuration (Configuration.java:loadProperty(2510)) - file:/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/tmp/cascading/staging/soberoi90064939/.staging/job_local90064939_0018/job.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.retry.interval;  Ignoring.
2015-02-12 16:12:50,441 WARN  conf.Configuration (Configuration.java:loadProperty(2510)) - file:/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/tmp/cascading/staging/soberoi90064939/.staging/job_local90064939_0018/job.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.attempts;  Ignoring.
2015-02-12 16:12:50,475 WARN  conf.Configuration (Configuration.java:loadProperty(2510)) - file:/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/test/tmp/mapred/local/localRunner/soberoi/job_local90064939_0018/job_local90064939_0018.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.retry.interval;  Ignoring.
2015-02-12 16:12:50,477 WARN  conf.Configuration (Configuration.java:loadProperty(2510)) - file:/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/test/tmp/mapred/local/localRunner/soberoi/job_local90064939_0018/job_local90064939_0018.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.attempts;  Ignoring.
2015-02-12 16:12:50,477 INFO  mapreduce.Job (Job.java:submit(1299)) - The url to track the job: http://localhost:8080/
2015-02-12 16:12:50,478 INFO  mapred.LocalJobRunner (LocalJobRunner.java:createOutputCommitter(471)) - OutputCommitter set in config null
2015-02-12 16:12:50,478 INFO  mapred.LocalJobRunner (LocalJobRunner.java:createOutputCommitter(489)) - OutputCommitter is org.apache.hadoop.mapred.FileOutputCommitter
2015-02-12 16:12:50,478 INFO  flow.FlowStep (BaseFlowStep.java:logInfo(834)) - [shape] submitted hadoop job: job_local90064939_0018
2015-02-12 16:12:50,478 INFO  flow.FlowStep (BaseFlowStep.java:logInfo(834)) - [shape] tracking url: http://localhost:8080/
2015-02-12 16:12:50,480 INFO  mapred.LocalJobRunner (LocalJobRunner.java:runTasks(448)) - Waiting for map tasks
2015-02-12 16:12:50,480 INFO  mapred.LocalJobRunner (LocalJobRunner.java:run(224)) - Starting task: attempt_local90064939_0018_m_000000_0
2015-02-12 16:12:50,481 INFO  util.ProcfsBasedProcessTree (ProcfsBasedProcessTree.java:isAvailable(181)) - ProcfsBasedProcessTree currently is supported only on Linux.
2015-02-12 16:12:50,481 INFO  mapred.Task (Task.java:initialize(587)) -  Using ResourceCalculatorProcessTree : null
2015-02-12 16:12:50,482 INFO  io.MultiInputSplit (MultiInputSplit.java:readFields(161)) - current split input path: file:/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/classes/test/data/lower.txt
2015-02-12 16:12:50,482 INFO  mapred.MapTask (MapTask.java:updateJobWithSplit(482)) - Processing split: cascading.tap.hadoop.io.MultiInputSplit@4dbe4157
2015-02-12 16:12:50,483 INFO  mapred.MapTask (MapTask.java:runOldMapper(436)) - numReduceTasks: 0
2015-02-12 16:12:50,488 INFO  hadoop.FlowMapper (FlowMapper.java:configure(71)) - cascading version: 2.6.3
2015-02-12 16:12:50,488 INFO  hadoop.FlowMapper (FlowMapper.java:configure(72)) - child jvm opts: -Xmx200m
2015-02-12 16:12:50,491 INFO  hadoop.FlowMapper (FlowMapper.java:configure(87)) - sourcing from: Hfs["TextLine[['offset', 'line']->[ALL]]"]["/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/classes/test/data/lower.txt"]
2015-02-12 16:12:50,491 INFO  hadoop.FlowMapper (FlowMapper.java:configure(90)) - sinking to: Hfs["TextLine[['line']->['item', 'element']]"]["/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/test/output/hadoop2-mr1/assemblyhelpersplatform/rename"]
2015-02-12 16:12:50,492 INFO  mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) - 
2015-02-12 16:12:50,493 INFO  mapred.Task (Task.java:done(1001)) - Task:attempt_local90064939_0018_m_000000_0 is done. And is in the process of committing
2015-02-12 16:12:50,494 INFO  mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) - 
2015-02-12 16:12:50,494 INFO  mapred.Task (Task.java:commit(1162)) - Task attempt_local90064939_0018_m_000000_0 is allowed to commit now
2015-02-12 16:12:50,494 INFO  output.FileOutputCommitter (FileOutputCommitter.java:commitTask(439)) - Saved output of task 'attempt_local90064939_0018_m_000000_0' to file:/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/test/output/hadoop2-mr1/assemblyhelpersplatform/rename/_temporary/0/task_local90064939_0018_m_000000
2015-02-12 16:12:50,495 INFO  mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) - file:/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/classes/test/data/lower.txt:0+19
2015-02-12 16:12:50,495 INFO  mapred.Task (Task.java:sendDone(1121)) - Task 'attempt_local90064939_0018_m_000000_0' done.
2015-02-12 16:12:50,496 INFO  mapred.LocalJobRunner (LocalJobRunner.java:run(249)) - Finishing task: attempt_local90064939_0018_m_000000_0
2015-02-12 16:12:50,496 INFO  mapred.LocalJobRunner (LocalJobRunner.java:runTasks(456)) - map task executor complete.
2015-02-12 16:12:50,504 INFO  util.Hadoop18TapUtil (Hadoop18TapUtil.java:cleanTempPath(219)) - deleting temp path /Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/test/output/hadoop2-mr1/assemblyhelpersplatform/rename/_temporary
2015-02-12 16:12:50,524 INFO  mapred.FileInputFormat (FileInputFormat.java:listStatus(247)) - Total input paths to process : 1
2015-02-12 16:12:50,525 INFO  cascading.PlatformTestCase (PlatformTestCase.java:tearDown(209)) - copying to local /Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/test/output//hadoop2-mr1/assemblyhelpersplatform/rename
2015-02-12 16:12:50,538 INFO  util.HadoopUtil (HadoopUtil.java:findMainClass(336)) - using default application jar, may cause class not found exceptions on the cluster
2015-02-12 16:12:50,539 INFO  planner.HadoopPlanner (HadoopPlanner.java:initialize(225)) - using application jar: /Users/soberoi/.gradle/caches/modules-2/files-2.1/cascading/cascading-hadoop2-mr1/2.6.3/9ae016e2b8d246b9c1de9168e715d9f1b971aa4a/cascading-hadoop2-mr1-2.6.3.jar
2015-02-12 16:12:50,547 INFO  Configuration.deprecation (Configuration.java:warnOnceIfDeprecated(1022)) - mapred.used.genericoptionsparser is deprecated. Instead, use mapreduce.client.genericoptionsparser.used
2015-02-12 16:12:50,548 INFO  Configuration.deprecation (Configuration.java:warnOnceIfDeprecated(1022)) - mapred.job.tracker is deprecated. Instead, use mapreduce.jobtracker.address
2015-02-12 16:12:50,549 INFO  Configuration.deprecation (Configuration.java:warnOnceIfDeprecated(1022)) - mapred.output.compress is deprecated. Instead, use mapreduce.output.fileoutputformat.compress
2015-02-12 16:12:50,556 INFO  flow.Flow (BaseFlow.java:logInfo(1378)) - [name] starting
2015-02-12 16:12:50,557 INFO  flow.Flow (BaseFlow.java:logInfo(1378)) - [name]  source: Hfs["TextDelimited[['num', 'char']]"]["/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/classes/test/data/rhs.txt"]
2015-02-12 16:12:50,557 INFO  flow.Flow (BaseFlow.java:logInfo(1378)) - [name]  source: Hfs["TextDelimited[['num', 'char']]"]["/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/classes/test/data/lhs.txt"]
2015-02-12 16:12:50,557 INFO  flow.Flow (BaseFlow.java:logInfo(1378)) - [name]  sink: Hfs["TextDelimited[['char', 'sum', 'count', 'average']]"]["/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/test/output/hadoop2-mr1/assemblyhelpersplatform/multimerge+true"]
2015-02-12 16:12:50,557 INFO  flow.Flow (BaseFlow.java:logInfo(1378)) - [name]  parallel execution is enabled: false
2015-02-12 16:12:50,557 INFO  flow.Flow (BaseFlow.java:logInfo(1378)) - [name]  starting jobs: 1
2015-02-12 16:12:50,557 INFO  flow.Flow (BaseFlow.java:logInfo(1378)) - [name]  allocating threads: 1
2015-02-12 16:12:50,557 INFO  flow.FlowStep (BaseFlowStep.java:logInfo(834)) - [name] starting step: (1/1) ...splatform/multimerge+true
2015-02-12 16:12:50,558 INFO  jvm.JvmMetrics (JvmMetrics.java:init(71)) - Cannot initialize JVM Metrics with processName=JobTracker, sessionId= - already initialized
2015-02-12 16:12:50,559 INFO  jvm.JvmMetrics (JvmMetrics.java:init(71)) - Cannot initialize JVM Metrics with processName=JobTracker, sessionId= - already initialized
2015-02-12 16:12:50,578 INFO  mapred.FileInputFormat (FileInputFormat.java:listStatus(247)) - Total input paths to process : 1
2015-02-12 16:12:50,579 INFO  mapred.FileInputFormat (FileInputFormat.java:listStatus(247)) - Total input paths to process : 1
2015-02-12 16:12:50,594 INFO  mapreduce.JobSubmitter (JobSubmitter.java:submitJobInternal(492)) - number of splits:2
2015-02-12 16:12:50,605 INFO  mapreduce.JobSubmitter (JobSubmitter.java:printTokens(575)) - Submitting tokens for job: job_local366847577_0019
2015-02-12 16:12:50,615 WARN  conf.Configuration (Configuration.java:loadProperty(2510)) - file:/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/tmp/cascading/staging/soberoi366847577/.staging/job_local366847577_0019/job.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.retry.interval;  Ignoring.
2015-02-12 16:12:50,616 WARN  conf.Configuration (Configuration.java:loadProperty(2510)) - file:/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/tmp/cascading/staging/soberoi366847577/.staging/job_local366847577_0019/job.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.attempts;  Ignoring.
2015-02-12 16:12:50,644 WARN  conf.Configuration (Configuration.java:loadProperty(2510)) - file:/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/test/tmp/mapred/local/localRunner/soberoi/job_local366847577_0019/job_local366847577_0019.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.retry.interval;  Ignoring.
2015-02-12 16:12:50,645 WARN  conf.Configuration (Configuration.java:loadProperty(2510)) - file:/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/test/tmp/mapred/local/localRunner/soberoi/job_local366847577_0019/job_local366847577_0019.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.attempts;  Ignoring.
2015-02-12 16:12:50,645 INFO  mapreduce.Job (Job.java:submit(1299)) - The url to track the job: http://localhost:8080/
2015-02-12 16:12:50,646 INFO  mapred.LocalJobRunner (LocalJobRunner.java:createOutputCommitter(471)) - OutputCommitter set in config null
2015-02-12 16:12:50,646 INFO  mapred.LocalJobRunner (LocalJobRunner.java:createOutputCommitter(489)) - OutputCommitter is org.apache.hadoop.mapred.FileOutputCommitter
2015-02-12 16:12:50,646 INFO  flow.FlowStep (BaseFlowStep.java:logInfo(834)) - [name] submitted hadoop job: job_local366847577_0019
2015-02-12 16:12:50,646 INFO  flow.FlowStep (BaseFlowStep.java:logInfo(834)) - [name] tracking url: http://localhost:8080/
2015-02-12 16:12:50,648 INFO  mapred.LocalJobRunner (LocalJobRunner.java:runTasks(448)) - Waiting for map tasks
2015-02-12 16:12:50,648 INFO  mapred.LocalJobRunner (LocalJobRunner.java:run(224)) - Starting task: attempt_local366847577_0019_m_000000_0
2015-02-12 16:12:50,649 INFO  util.ProcfsBasedProcessTree (ProcfsBasedProcessTree.java:isAvailable(181)) - ProcfsBasedProcessTree currently is supported only on Linux.
2015-02-12 16:12:50,649 INFO  mapred.Task (Task.java:initialize(587)) -  Using ResourceCalculatorProcessTree : null
2015-02-12 16:12:50,650 INFO  io.MultiInputSplit (MultiInputSplit.java:readFields(161)) - current split input path: file:/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/classes/test/data/lhs.txt
2015-02-12 16:12:50,650 INFO  mapred.MapTask (MapTask.java:updateJobWithSplit(482)) - Processing split: cascading.tap.hadoop.io.MultiInputSplit@3500b5d7
2015-02-12 16:12:50,651 INFO  mapred.MapTask (MapTask.java:runOldMapper(436)) - numReduceTasks: 1
2015-02-12 16:12:50,667 INFO  mapred.MapTask (MapTask.java:setEquator(1202)) - (EQUATOR) 0 kvi 26214396(104857584)
2015-02-12 16:12:50,668 INFO  mapred.MapTask (MapTask.java:init(995)) - mapreduce.task.io.sort.mb: 100
2015-02-12 16:12:50,668 INFO  mapred.MapTask (MapTask.java:init(996)) - soft limit at 83886080
2015-02-12 16:12:50,668 INFO  mapred.MapTask (MapTask.java:init(997)) - bufstart = 0; bufvoid = 104857600
2015-02-12 16:12:50,668 INFO  mapred.MapTask (MapTask.java:init(998)) - kvstart = 26214396; length = 6553600
2015-02-12 16:12:50,669 INFO  mapred.MapTask (MapTask.java:createSortingCollector(402)) - Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer
2015-02-12 16:12:50,669 INFO  hadoop.FlowMapper (FlowMapper.java:configure(71)) - cascading version: 2.6.3
2015-02-12 16:12:50,669 INFO  hadoop.FlowMapper (FlowMapper.java:configure(72)) - child jvm opts: -Xmx200m
2015-02-12 16:12:50,676 INFO  hadoop.FlowMapper (FlowMapper.java:configure(87)) - sourcing from: Hfs["TextDelimited[['num', 'char']]"]["/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/classes/test/data/lhs.txt"]
2015-02-12 16:12:50,676 INFO  hadoop.FlowMapper (FlowMapper.java:configure(90)) - sinking to: GroupBy(name)[by:[{1}:'char']]
2015-02-12 16:12:50,676 INFO  assembly.AggregateBy (AggregateBy.java:prepare(267)) - using threshold value: 2
2015-02-12 16:12:50,677 INFO  assembly.AggregateBy (AggregateBy.java:removeEldestEntry(327)) - flushed keys num times: 1, with threshold: 2
2015-02-12 16:12:50,677 INFO  assembly.AggregateBy (AggregateBy.java:removeEldestEntry(328)) - mem on flush (mb), free: 195, total: 509, max: 672
2015-02-12 16:12:50,677 INFO  assembly.AggregateBy (AggregateBy.java:removeEldestEntry(333)) - total mem is 75% of max mem, to better utilize unused memory consider increasing current LRU threshold with system property "cascading.aggregateby.threshold"
2015-02-12 16:12:50,678 INFO  assembly.AggregateBy (AggregateBy.java:removeEldestEntry(327)) - flushed keys num times: 3, with threshold: 2
2015-02-12 16:12:50,678 INFO  assembly.AggregateBy (AggregateBy.java:removeEldestEntry(328)) - mem on flush (mb), free: 195, total: 509, max: 672
2015-02-12 16:12:50,678 INFO  assembly.AggregateBy (AggregateBy.java:removeEldestEntry(333)) - total mem is 75% of max mem, to better utilize unused memory consider increasing current LRU threshold with system property "cascading.aggregateby.threshold"
2015-02-12 16:12:50,679 INFO  assembly.AggregateBy (AggregateBy.java:removeEldestEntry(327)) - flushed keys num times: 5, with threshold: 2
2015-02-12 16:12:50,679 INFO  assembly.AggregateBy (AggregateBy.java:removeEldestEntry(328)) - mem on flush (mb), free: 195, total: 509, max: 672
2015-02-12 16:12:50,679 INFO  assembly.AggregateBy (AggregateBy.java:removeEldestEntry(333)) - total mem is 75% of max mem, to better utilize unused memory consider increasing current LRU threshold with system property "cascading.aggregateby.threshold"
2015-02-12 16:12:50,679 INFO  assembly.AggregateBy (AggregateBy.java:removeEldestEntry(327)) - flushed keys num times: 7, with threshold: 2
2015-02-12 16:12:50,679 INFO  assembly.AggregateBy (AggregateBy.java:removeEldestEntry(328)) - mem on flush (mb), free: 195, total: 509, max: 672
2015-02-12 16:12:50,680 INFO  assembly.AggregateBy (AggregateBy.java:removeEldestEntry(333)) - total mem is 75% of max mem, to better utilize unused memory consider increasing current LRU threshold with system property "cascading.aggregateby.threshold"
2015-02-12 16:12:50,681 INFO  mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) - 
2015-02-12 16:12:50,681 INFO  mapred.MapTask (MapTask.java:flush(1457)) - Starting flush of map output
2015-02-12 16:12:50,681 INFO  mapred.MapTask (MapTask.java:flush(1475)) - Spilling map output
2015-02-12 16:12:50,681 INFO  mapred.MapTask (MapTask.java:flush(1476)) - bufstart = 0; bufend = 216; bufvoid = 104857600
2015-02-12 16:12:50,681 INFO  mapred.MapTask (MapTask.java:flush(1478)) - kvstart = 26214396(104857584); kvend = 26214364(104857456); length = 33/6553600
2015-02-12 16:12:50,684 INFO  mapred.MapTask (MapTask.java:sortAndSpill(1660)) - Finished spill 0
2015-02-12 16:12:50,685 INFO  mapred.Task (Task.java:done(1001)) - Task:attempt_local366847577_0019_m_000000_0 is done. And is in the process of committing
2015-02-12 16:12:50,687 INFO  mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) - file:/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/classes/test/data/lhs.txt:0+52
2015-02-12 16:12:50,687 INFO  mapred.Task (Task.java:sendDone(1121)) - Task 'attempt_local366847577_0019_m_000000_0' done.
2015-02-12 16:12:50,687 INFO  mapred.LocalJobRunner (LocalJobRunner.java:run(249)) - Finishing task: attempt_local366847577_0019_m_000000_0
2015-02-12 16:12:50,687 INFO  mapred.LocalJobRunner (LocalJobRunner.java:run(224)) - Starting task: attempt_local366847577_0019_m_000001_0
2015-02-12 16:12:50,688 INFO  util.ProcfsBasedProcessTree (ProcfsBasedProcessTree.java:isAvailable(181)) - ProcfsBasedProcessTree currently is supported only on Linux.
2015-02-12 16:12:50,688 INFO  mapred.Task (Task.java:initialize(587)) -  Using ResourceCalculatorProcessTree : null
2015-02-12 16:12:50,689 INFO  io.MultiInputSplit (MultiInputSplit.java:readFields(161)) - current split input path: file:/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/classes/test/data/rhs.txt
2015-02-12 16:12:50,689 INFO  mapred.MapTask (MapTask.java:updateJobWithSplit(482)) - Processing split: cascading.tap.hadoop.io.MultiInputSplit@59f4e49e
2015-02-12 16:12:50,690 INFO  mapred.MapTask (MapTask.java:runOldMapper(436)) - numReduceTasks: 1
2015-02-12 16:12:50,701 INFO  mapred.MapTask (MapTask.java:setEquator(1202)) - (EQUATOR) 0 kvi 26214396(104857584)
2015-02-12 16:12:50,701 INFO  mapred.MapTask (MapTask.java:init(995)) - mapreduce.task.io.sort.mb: 100
2015-02-12 16:12:50,702 INFO  mapred.MapTask (MapTask.java:init(996)) - soft limit at 83886080
2015-02-12 16:12:50,702 INFO  mapred.MapTask (MapTask.java:init(997)) - bufstart = 0; bufvoid = 104857600
2015-02-12 16:12:50,702 INFO  mapred.MapTask (MapTask.java:init(998)) - kvstart = 26214396; length = 6553600
2015-02-12 16:12:50,703 INFO  mapred.MapTask (MapTask.java:createSortingCollector(402)) - Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer
2015-02-12 16:12:50,703 INFO  hadoop.FlowMapper (FlowMapper.java:configure(71)) - cascading version: 2.6.3
2015-02-12 16:12:50,703 INFO  hadoop.FlowMapper (FlowMapper.java:configure(72)) - child jvm opts: -Xmx200m
2015-02-12 16:12:50,709 INFO  hadoop.FlowMapper (FlowMapper.java:configure(87)) - sourcing from: Hfs["TextDelimited[['num', 'char']]"]["/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/classes/test/data/rhs.txt"]
2015-02-12 16:12:50,709 INFO  hadoop.FlowMapper (FlowMapper.java:configure(90)) - sinking to: GroupBy(name)[by:[{1}:'char']]
2015-02-12 16:12:50,709 INFO  assembly.AggregateBy (AggregateBy.java:prepare(267)) - using threshold value: 2
2015-02-12 16:12:50,712 INFO  assembly.AggregateBy (AggregateBy.java:removeEldestEntry(327)) - flushed keys num times: 1, with threshold: 2
2015-02-12 16:12:50,712 INFO  assembly.AggregateBy (AggregateBy.java:removeEldestEntry(328)) - mem on flush (mb), free: 93, total: 509, max: 672
2015-02-12 16:12:50,712 INFO  assembly.AggregateBy (AggregateBy.java:removeEldestEntry(333)) - total mem is 75% of max mem, to better utilize unused memory consider increasing current LRU threshold with system property "cascading.aggregateby.threshold"
2015-02-12 16:12:50,713 INFO  assembly.AggregateBy (AggregateBy.java:removeEldestEntry(327)) - flushed keys num times: 3, with threshold: 2
2015-02-12 16:12:50,713 INFO  assembly.AggregateBy (AggregateBy.java:removeEldestEntry(328)) - mem on flush (mb), free: 93, total: 509, max: 672
2015-02-12 16:12:50,713 INFO  assembly.AggregateBy (AggregateBy.java:removeEldestEntry(333)) - total mem is 75% of max mem, to better utilize unused memory consider increasing current LRU threshold with system property "cascading.aggregateby.threshold"
2015-02-12 16:12:50,713 INFO  assembly.AggregateBy (AggregateBy.java:removeEldestEntry(327)) - flushed keys num times: 5, with threshold: 2
2015-02-12 16:12:50,713 INFO  assembly.AggregateBy (AggregateBy.java:removeEldestEntry(328)) - mem on flush (mb), free: 93, total: 509, max: 672
2015-02-12 16:12:50,713 INFO  assembly.AggregateBy (AggregateBy.java:removeEldestEntry(333)) - total mem is 75% of max mem, to better utilize unused memory consider increasing current LRU threshold with system property "cascading.aggregateby.threshold"
2015-02-12 16:12:50,714 INFO  assembly.AggregateBy (AggregateBy.java:removeEldestEntry(327)) - flushed keys num times: 7, with threshold: 2
2015-02-12 16:12:50,714 INFO  assembly.AggregateBy (AggregateBy.java:removeEldestEntry(328)) - mem on flush (mb), free: 93, total: 509, max: 672
2015-02-12 16:12:50,714 INFO  assembly.AggregateBy (AggregateBy.java:removeEldestEntry(333)) - total mem is 75% of max mem, to better utilize unused memory consider increasing current LRU threshold with system property "cascading.aggregateby.threshold"
2015-02-12 16:12:50,715 INFO  mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) - 
2015-02-12 16:12:50,715 INFO  mapred.MapTask (MapTask.java:flush(1457)) - Starting flush of map output
2015-02-12 16:12:50,715 INFO  mapred.MapTask (MapTask.java:flush(1475)) - Spilling map output
2015-02-12 16:12:50,715 INFO  mapred.MapTask (MapTask.java:flush(1476)) - bufstart = 0; bufend = 216; bufvoid = 104857600
2015-02-12 16:12:50,715 INFO  mapred.MapTask (MapTask.java:flush(1478)) - kvstart = 26214396(104857584); kvend = 26214364(104857456); length = 33/6553600
2015-02-12 16:12:50,717 INFO  mapred.MapTask (MapTask.java:sortAndSpill(1660)) - Finished spill 0
2015-02-12 16:12:50,718 INFO  mapred.Task (Task.java:done(1001)) - Task:attempt_local366847577_0019_m_000001_0 is done. And is in the process of committing
2015-02-12 16:12:50,719 INFO  mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) - file:/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/classes/test/data/rhs.txt:0+51
2015-02-12 16:12:50,720 INFO  mapred.Task (Task.java:sendDone(1121)) - Task 'attempt_local366847577_0019_m_000001_0' done.
2015-02-12 16:12:50,720 INFO  mapred.LocalJobRunner (LocalJobRunner.java:run(249)) - Finishing task: attempt_local366847577_0019_m_000001_0
2015-02-12 16:12:50,720 INFO  mapred.LocalJobRunner (LocalJobRunner.java:runTasks(456)) - map task executor complete.
2015-02-12 16:12:50,720 INFO  mapred.LocalJobRunner (LocalJobRunner.java:runTasks(448)) - Waiting for reduce tasks
2015-02-12 16:12:50,720 INFO  mapred.LocalJobRunner (LocalJobRunner.java:run(302)) - Starting task: attempt_local366847577_0019_r_000000_0
2015-02-12 16:12:50,721 INFO  util.ProcfsBasedProcessTree (ProcfsBasedProcessTree.java:isAvailable(181)) - ProcfsBasedProcessTree currently is supported only on Linux.
2015-02-12 16:12:50,721 INFO  mapred.Task (Task.java:initialize(587)) -  Using ResourceCalculatorProcessTree : null
2015-02-12 16:12:50,721 INFO  mapred.ReduceTask (ReduceTask.java:run(362)) - Using ShuffleConsumerPlugin: org.apache.hadoop.mapreduce.task.reduce.Shuffle@497acafa
2015-02-12 16:12:50,722 INFO  reduce.MergeManagerImpl (MergeManagerImpl.java:<init>(195)) - MergerManager: memoryLimit=493250144, maxSingleShuffleLimit=123312536, mergeThreshold=325545120, ioSortFactor=10, memToMemMergeOutputsThreshold=10
2015-02-12 16:12:50,722 INFO  reduce.EventFetcher (EventFetcher.java:run(61)) - attempt_local366847577_0019_r_000000_0 Thread started: EventFetcher for fetching Map Completion Events
2015-02-12 16:12:50,723 INFO  reduce.LocalFetcher (LocalFetcher.java:copyMapOutput(141)) - localfetcher#15 about to shuffle output of map attempt_local366847577_0019_m_000000_0 decomp: 236 len: 240 to MEMORY
2015-02-12 16:12:50,724 INFO  reduce.InMemoryMapOutput (InMemoryMapOutput.java:shuffle(100)) - Read 236 bytes from map-output for attempt_local366847577_0019_m_000000_0
2015-02-12 16:12:50,724 INFO  reduce.MergeManagerImpl (MergeManagerImpl.java:closeInMemoryFile(313)) - closeInMemoryFile -> map-output of size: 236, inMemoryMapOutputs.size() -> 1, commitMemory -> 0, usedMemory ->236
2015-02-12 16:12:50,725 INFO  reduce.LocalFetcher (LocalFetcher.java:copyMapOutput(141)) - localfetcher#15 about to shuffle output of map attempt_local366847577_0019_m_000001_0 decomp: 236 len: 240 to MEMORY
2015-02-12 16:12:50,725 INFO  reduce.InMemoryMapOutput (InMemoryMapOutput.java:shuffle(100)) - Read 236 bytes from map-output for attempt_local366847577_0019_m_000001_0
2015-02-12 16:12:50,725 INFO  reduce.MergeManagerImpl (MergeManagerImpl.java:closeInMemoryFile(313)) - closeInMemoryFile -> map-output of size: 236, inMemoryMapOutputs.size() -> 2, commitMemory -> 236, usedMemory ->472
2015-02-12 16:12:50,725 INFO  reduce.EventFetcher (EventFetcher.java:run(76)) - EventFetcher is interrupted.. Returning
2015-02-12 16:12:50,726 INFO  mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) - 2 / 2 copied.
2015-02-12 16:12:50,726 INFO  reduce.MergeManagerImpl (MergeManagerImpl.java:finalMerge(673)) - finalMerge called with 2 in-memory map-outputs and 0 on-disk map-outputs
2015-02-12 16:12:50,728 INFO  mapred.Merger (Merger.java:merge(606)) - Merging 2 sorted segments
2015-02-12 16:12:50,728 INFO  mapred.Merger (Merger.java:merge(705)) - Down to the last merge-pass, with 2 segments left of total size: 454 bytes
2015-02-12 16:12:50,730 INFO  reduce.MergeManagerImpl (MergeManagerImpl.java:finalMerge(750)) - Merged 2 segments, 472 bytes to disk to satisfy reduce memory limit
2015-02-12 16:12:50,730 INFO  reduce.MergeManagerImpl (MergeManagerImpl.java:finalMerge(780)) - Merging 1 files, 474 bytes from disk
2015-02-12 16:12:50,730 INFO  reduce.MergeManagerImpl (MergeManagerImpl.java:finalMerge(795)) - Merging 0 segments, 0 bytes from memory into reduce
2015-02-12 16:12:50,730 INFO  mapred.Merger (Merger.java:merge(606)) - Merging 1 sorted segments
2015-02-12 16:12:50,731 INFO  mapred.Merger (Merger.java:merge(705)) - Down to the last merge-pass, with 1 segments left of total size: 461 bytes
2015-02-12 16:12:50,731 INFO  mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) - 2 / 2 copied.
2015-02-12 16:12:50,732 INFO  hadoop.FlowReducer (FlowReducer.java:configure(78)) - cascading version: 2.6.3
2015-02-12 16:12:50,732 INFO  hadoop.FlowReducer (FlowReducer.java:configure(79)) - child jvm opts: -Xmx200m
2015-02-12 16:12:50,737 INFO  hadoop.FlowReducer (FlowReducer.java:configure(97)) - sourcing from: GroupBy(name)[by:[{1}:'char']]
2015-02-12 16:12:50,737 INFO  hadoop.FlowReducer (FlowReducer.java:configure(100)) - sinking to: Hfs["TextDelimited[['char', 'sum', 'count', 'average']]"]["/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/test/output/hadoop2-mr1/assemblyhelpersplatform/multimerge+true"]
2015-02-12 16:12:50,746 INFO  mapred.Task (Task.java:done(1001)) - Task:attempt_local366847577_0019_r_000000_0 is done. And is in the process of committing
2015-02-12 16:12:50,747 INFO  mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) - 2 / 2 copied.
2015-02-12 16:12:50,747 INFO  mapred.Task (Task.java:commit(1162)) - Task attempt_local366847577_0019_r_000000_0 is allowed to commit now
2015-02-12 16:12:50,748 INFO  output.FileOutputCommitter (FileOutputCommitter.java:commitTask(439)) - Saved output of task 'attempt_local366847577_0019_r_000000_0' to file:/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/test/output/hadoop2-mr1/assemblyhelpersplatform/multimerge+true/_temporary/0/task_local366847577_0019_r_000000
2015-02-12 16:12:50,748 INFO  mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) - reduce > reduce
2015-02-12 16:12:50,748 INFO  mapred.Task (Task.java:sendDone(1121)) - Task 'attempt_local366847577_0019_r_000000_0' done.
2015-02-12 16:12:50,748 INFO  mapred.LocalJobRunner (LocalJobRunner.java:run(325)) - Finishing task: attempt_local366847577_0019_r_000000_0
2015-02-12 16:12:50,749 INFO  mapred.LocalJobRunner (LocalJobRunner.java:runTasks(456)) - reduce task executor complete.
2015-02-12 16:12:50,766 INFO  util.Hadoop18TapUtil (Hadoop18TapUtil.java:cleanTempPath(219)) - deleting temp path /Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/test/output/hadoop2-mr1/assemblyhelpersplatform/multimerge+true/_temporary
2015-02-12 16:12:50,801 INFO  mapred.FileInputFormat (FileInputFormat.java:listStatus(247)) - Total input paths to process : 1
2015-02-12 16:12:50,839 INFO  mapred.FileInputFormat (FileInputFormat.java:listStatus(247)) - Total input paths to process : 1
2015-02-12 16:12:50,841 INFO  cascading.PlatformTestCase (PlatformTestCase.java:tearDown(209)) - copying to local /Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/test/output//hadoop2-mr1/assemblyhelpersplatform/multimerge+true
2015-02-12 16:12:50,859 INFO  util.HadoopUtil (HadoopUtil.java:findMainClass(336)) - using default application jar, may cause class not found exceptions on the cluster
2015-02-12 16:12:50,860 INFO  planner.HadoopPlanner (HadoopPlanner.java:initialize(225)) - using application jar: /Users/soberoi/.gradle/caches/modules-2/files-2.1/cascading/cascading-hadoop2-mr1/2.6.3/9ae016e2b8d246b9c1de9168e715d9f1b971aa4a/cascading-hadoop2-mr1-2.6.3.jar
2015-02-12 16:12:50,871 INFO  Configuration.deprecation (Configuration.java:warnOnceIfDeprecated(1022)) - mapred.used.genericoptionsparser is deprecated. Instead, use mapreduce.client.genericoptionsparser.used
2015-02-12 16:12:50,873 INFO  Configuration.deprecation (Configuration.java:warnOnceIfDeprecated(1022)) - mapred.job.tracker is deprecated. Instead, use mapreduce.jobtracker.address
2015-02-12 16:12:50,878 INFO  Configuration.deprecation (Configuration.java:warnOnceIfDeprecated(1022)) - mapred.output.compress is deprecated. Instead, use mapreduce.output.fileoutputformat.compress
2015-02-12 16:12:50,884 INFO  flow.Flow (BaseFlow.java:logInfo(1378)) - [second] starting
2015-02-12 16:12:50,885 INFO  flow.Flow (BaseFlow.java:logInfo(1378)) - [second]  source: Hfs["TextDelimited[['num', 'char']]"]["/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/classes/test/data/lhs.txt"]
2015-02-12 16:12:50,885 INFO  flow.Flow (BaseFlow.java:logInfo(1378)) - [second]  sink: Hfs["TextDelimited[['char', 'count']]"]["/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/test/output/hadoop2-mr1/assemblyhelpersplatform/samesourcemergecount"]
2015-02-12 16:12:50,885 INFO  flow.Flow (BaseFlow.java:logInfo(1378)) - [second]  parallel execution is enabled: false
2015-02-12 16:12:50,885 INFO  flow.Flow (BaseFlow.java:logInfo(1378)) - [second]  starting jobs: 2
2015-02-12 16:12:50,885 INFO  flow.Flow (BaseFlow.java:logInfo(1378)) - [second]  allocating threads: 1
2015-02-12 16:12:50,885 INFO  flow.FlowStep (BaseFlowStep.java:logInfo(834)) - [second] starting step: (1/2)
2015-02-12 16:12:50,886 INFO  jvm.JvmMetrics (JvmMetrics.java:init(71)) - Cannot initialize JVM Metrics with processName=JobTracker, sessionId= - already initialized
2015-02-12 16:12:50,887 INFO  jvm.JvmMetrics (JvmMetrics.java:init(71)) - Cannot initialize JVM Metrics with processName=JobTracker, sessionId= - already initialized
2015-02-12 16:12:50,908 INFO  mapred.FileInputFormat (FileInputFormat.java:listStatus(247)) - Total input paths to process : 1
2015-02-12 16:12:50,924 INFO  mapreduce.JobSubmitter (JobSubmitter.java:submitJobInternal(492)) - number of splits:1
2015-02-12 16:12:50,934 INFO  mapreduce.JobSubmitter (JobSubmitter.java:printTokens(575)) - Submitting tokens for job: job_local693942087_0020
2015-02-12 16:12:50,943 WARN  conf.Configuration (Configuration.java:loadProperty(2510)) - file:/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/tmp/cascading/staging/soberoi693942087/.staging/job_local693942087_0020/job.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.retry.interval;  Ignoring.
2015-02-12 16:12:50,944 WARN  conf.Configuration (Configuration.java:loadProperty(2510)) - file:/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/tmp/cascading/staging/soberoi693942087/.staging/job_local693942087_0020/job.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.attempts;  Ignoring.
2015-02-12 16:12:50,973 WARN  conf.Configuration (Configuration.java:loadProperty(2510)) - file:/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/test/tmp/mapred/local/localRunner/soberoi/job_local693942087_0020/job_local693942087_0020.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.retry.interval;  Ignoring.
2015-02-12 16:12:50,975 WARN  conf.Configuration (Configuration.java:loadProperty(2510)) - file:/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/test/tmp/mapred/local/localRunner/soberoi/job_local693942087_0020/job_local693942087_0020.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.attempts;  Ignoring.
2015-02-12 16:12:50,975 INFO  mapreduce.Job (Job.java:submit(1299)) - The url to track the job: http://localhost:8080/
2015-02-12 16:12:50,976 INFO  mapred.LocalJobRunner (LocalJobRunner.java:createOutputCommitter(471)) - OutputCommitter set in config null
2015-02-12 16:12:50,976 INFO  mapred.LocalJobRunner (LocalJobRunner.java:createOutputCommitter(489)) - OutputCommitter is org.apache.hadoop.mapred.FileOutputCommitter
2015-02-12 16:12:50,976 INFO  flow.FlowStep (BaseFlowStep.java:logInfo(834)) - [second] submitted hadoop job: job_local693942087_0020
2015-02-12 16:12:50,977 INFO  flow.FlowStep (BaseFlowStep.java:logInfo(834)) - [second] tracking url: http://localhost:8080/
2015-02-12 16:12:50,978 INFO  mapred.LocalJobRunner (LocalJobRunner.java:runTasks(448)) - Waiting for map tasks
2015-02-12 16:12:50,978 INFO  mapred.LocalJobRunner (LocalJobRunner.java:run(224)) - Starting task: attempt_local693942087_0020_m_000000_0
2015-02-12 16:12:50,979 INFO  util.ProcfsBasedProcessTree (ProcfsBasedProcessTree.java:isAvailable(181)) - ProcfsBasedProcessTree currently is supported only on Linux.
2015-02-12 16:12:50,979 INFO  mapred.Task (Task.java:initialize(587)) -  Using ResourceCalculatorProcessTree : null
2015-02-12 16:12:50,980 INFO  io.MultiInputSplit (MultiInputSplit.java:readFields(161)) - current split input path: file:/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/classes/test/data/lhs.txt
2015-02-12 16:12:50,980 INFO  mapred.MapTask (MapTask.java:updateJobWithSplit(482)) - Processing split: cascading.tap.hadoop.io.MultiInputSplit@4361a2e6
2015-02-12 16:12:50,980 INFO  mapred.MapTask (MapTask.java:runOldMapper(436)) - numReduceTasks: 1
2015-02-12 16:12:50,994 INFO  mapred.MapTask (MapTask.java:setEquator(1202)) - (EQUATOR) 0 kvi 26214396(104857584)
2015-02-12 16:12:50,994 INFO  mapred.MapTask (MapTask.java:init(995)) - mapreduce.task.io.sort.mb: 100
2015-02-12 16:12:50,994 INFO  mapred.MapTask (MapTask.java:init(996)) - soft limit at 83886080
2015-02-12 16:12:50,994 INFO  mapred.MapTask (MapTask.java:init(997)) - bufstart = 0; bufvoid = 104857600
2015-02-12 16:12:50,995 INFO  mapred.MapTask (MapTask.java:init(998)) - kvstart = 26214396; length = 6553600
2015-02-12 16:12:50,996 INFO  mapred.MapTask (MapTask.java:createSortingCollector(402)) - Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer
2015-02-12 16:12:50,996 INFO  hadoop.FlowMapper (FlowMapper.java:configure(71)) - cascading version: 2.6.3
2015-02-12 16:12:50,996 INFO  hadoop.FlowMapper (FlowMapper.java:configure(72)) - child jvm opts: -Xmx200m
2015-02-12 16:12:51,005 INFO  hadoop.FlowMapper (FlowMapper.java:configure(87)) - sourcing from: Hfs["TextDelimited[['num', 'char']]"]["/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/classes/test/data/lhs.txt"]
2015-02-12 16:12:51,006 INFO  hadoop.FlowMapper (FlowMapper.java:configure(90)) - sinking to: GroupBy(first)[by:[{1}:'char']]
2015-02-12 16:12:51,006 INFO  assembly.AggregateBy (AggregateBy.java:prepare(267)) - using threshold value: 2
2015-02-12 16:12:51,009 INFO  assembly.AggregateBy (AggregateBy.java:removeEldestEntry(327)) - flushed keys num times: 1, with threshold: 2
2015-02-12 16:12:51,009 INFO  assembly.AggregateBy (AggregateBy.java:removeEldestEntry(328)) - mem on flush (mb), free: 155, total: 510, max: 672
2015-02-12 16:12:51,009 INFO  assembly.AggregateBy (AggregateBy.java:removeEldestEntry(333)) - total mem is 75% of max mem, to better utilize unused memory consider increasing current LRU threshold with system property "cascading.aggregateby.threshold"
2015-02-12 16:12:51,010 INFO  assembly.AggregateBy (AggregateBy.java:removeEldestEntry(327)) - flushed keys num times: 3, with threshold: 2
2015-02-12 16:12:51,010 INFO  assembly.AggregateBy (AggregateBy.java:removeEldestEntry(328)) - mem on flush (mb), free: 155, total: 510, max: 672
2015-02-12 16:12:51,010 INFO  assembly.AggregateBy (AggregateBy.java:removeEldestEntry(333)) - total mem is 75% of max mem, to better utilize unused memory consider increasing current LRU threshold with system property "cascading.aggregateby.threshold"
2015-02-12 16:12:51,010 INFO  assembly.AggregateBy (AggregateBy.java:removeEldestEntry(327)) - flushed keys num times: 5, with threshold: 2
2015-02-12 16:12:51,011 INFO  assembly.AggregateBy (AggregateBy.java:removeEldestEntry(328)) - mem on flush (mb), free: 155, total: 510, max: 672
2015-02-12 16:12:51,011 INFO  assembly.AggregateBy (AggregateBy.java:removeEldestEntry(333)) - total mem is 75% of max mem, to better utilize unused memory consider increasing current LRU threshold with system property "cascading.aggregateby.threshold"
2015-02-12 16:12:51,011 INFO  assembly.AggregateBy (AggregateBy.java:removeEldestEntry(327)) - flushed keys num times: 7, with threshold: 2
2015-02-12 16:12:51,011 INFO  assembly.AggregateBy (AggregateBy.java:removeEldestEntry(328)) - mem on flush (mb), free: 155, total: 510, max: 672
2015-02-12 16:12:51,011 INFO  assembly.AggregateBy (AggregateBy.java:removeEldestEntry(333)) - total mem is 75% of max mem, to better utilize unused memory consider increasing current LRU threshold with system property "cascading.aggregateby.threshold"
2015-02-12 16:12:51,012 INFO  mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) - 
2015-02-12 16:12:51,012 INFO  mapred.MapTask (MapTask.java:flush(1457)) - Starting flush of map output
2015-02-12 16:12:51,012 INFO  mapred.MapTask (MapTask.java:flush(1475)) - Spilling map output
2015-02-12 16:12:51,012 INFO  mapred.MapTask (MapTask.java:flush(1476)) - bufstart = 0; bufend = 99; bufvoid = 104857600
2015-02-12 16:12:51,013 INFO  mapred.MapTask (MapTask.java:flush(1478)) - kvstart = 26214396(104857584); kvend = 26214364(104857456); length = 33/6553600
2015-02-12 16:12:51,014 INFO  mapred.MapTask (MapTask.java:sortAndSpill(1660)) - Finished spill 0
2015-02-12 16:12:51,016 INFO  mapred.Task (Task.java:done(1001)) - Task:attempt_local693942087_0020_m_000000_0 is done. And is in the process of committing
2015-02-12 16:12:51,017 INFO  mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) - file:/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/classes/test/data/lhs.txt:0+52
2015-02-12 16:12:51,017 INFO  mapred.Task (Task.java:sendDone(1121)) - Task 'attempt_local693942087_0020_m_000000_0' done.
2015-02-12 16:12:51,017 INFO  mapred.LocalJobRunner (LocalJobRunner.java:run(249)) - Finishing task: attempt_local693942087_0020_m_000000_0
2015-02-12 16:12:51,017 INFO  mapred.LocalJobRunner (LocalJobRunner.java:runTasks(456)) - map task executor complete.
2015-02-12 16:12:51,018 INFO  mapred.LocalJobRunner (LocalJobRunner.java:runTasks(448)) - Waiting for reduce tasks
2015-02-12 16:12:51,018 INFO  mapred.LocalJobRunner (LocalJobRunner.java:run(302)) - Starting task: attempt_local693942087_0020_r_000000_0
2015-02-12 16:12:51,019 INFO  util.ProcfsBasedProcessTree (ProcfsBasedProcessTree.java:isAvailable(181)) - ProcfsBasedProcessTree currently is supported only on Linux.
2015-02-12 16:12:51,019 INFO  mapred.Task (Task.java:initialize(587)) -  Using ResourceCalculatorProcessTree : null
2015-02-12 16:12:51,019 INFO  mapred.ReduceTask (ReduceTask.java:run(362)) - Using ShuffleConsumerPlugin: org.apache.hadoop.mapreduce.task.reduce.Shuffle@20155887
2015-02-12 16:12:51,020 INFO  reduce.MergeManagerImpl (MergeManagerImpl.java:<init>(195)) - MergerManager: memoryLimit=493250144, maxSingleShuffleLimit=123312536, mergeThreshold=325545120, ioSortFactor=10, memToMemMergeOutputsThreshold=10
2015-02-12 16:12:51,020 INFO  reduce.EventFetcher (EventFetcher.java:run(61)) - attempt_local693942087_0020_r_000000_0 Thread started: EventFetcher for fetching Map Completion Events
2015-02-12 16:12:51,021 INFO  reduce.LocalFetcher (LocalFetcher.java:copyMapOutput(141)) - localfetcher#16 about to shuffle output of map attempt_local693942087_0020_m_000000_0 decomp: 119 len: 123 to MEMORY
2015-02-12 16:12:51,021 INFO  reduce.InMemoryMapOutput (InMemoryMapOutput.java:shuffle(100)) - Read 119 bytes from map-output for attempt_local693942087_0020_m_000000_0
2015-02-12 16:12:51,021 INFO  reduce.MergeManagerImpl (MergeManagerImpl.java:closeInMemoryFile(313)) - closeInMemoryFile -> map-output of size: 119, inMemoryMapOutputs.size() -> 1, commitMemory -> 0, usedMemory ->119
2015-02-12 16:12:51,022 INFO  reduce.EventFetcher (EventFetcher.java:run(76)) - EventFetcher is interrupted.. Returning
2015-02-12 16:12:51,022 INFO  mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) - 1 / 1 copied.
2015-02-12 16:12:51,022 INFO  reduce.MergeManagerImpl (MergeManagerImpl.java:finalMerge(673)) - finalMerge called with 1 in-memory map-outputs and 0 on-disk map-outputs
2015-02-12 16:12:51,023 INFO  mapred.Merger (Merger.java:merge(606)) - Merging 1 sorted segments
2015-02-12 16:12:51,023 INFO  mapred.Merger (Merger.java:merge(705)) - Down to the last merge-pass, with 1 segments left of total size: 110 bytes
2015-02-12 16:12:51,025 INFO  reduce.MergeManagerImpl (MergeManagerImpl.java:finalMerge(750)) - Merged 1 segments, 119 bytes to disk to satisfy reduce memory limit
2015-02-12 16:12:51,025 INFO  reduce.MergeManagerImpl (MergeManagerImpl.java:finalMerge(780)) - Merging 1 files, 123 bytes from disk
2015-02-12 16:12:51,025 INFO  reduce.MergeManagerImpl (MergeManagerImpl.java:finalMerge(795)) - Merging 0 segments, 0 bytes from memory into reduce
2015-02-12 16:12:51,025 INFO  mapred.Merger (Merger.java:merge(606)) - Merging 1 sorted segments
2015-02-12 16:12:51,026 INFO  mapred.Merger (Merger.java:merge(705)) - Down to the last merge-pass, with 1 segments left of total size: 110 bytes
2015-02-12 16:12:51,026 INFO  mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) - 1 / 1 copied.
2015-02-12 16:12:51,027 INFO  hadoop.FlowReducer (FlowReducer.java:configure(78)) - cascading version: 2.6.3
2015-02-12 16:12:51,027 INFO  hadoop.FlowReducer (FlowReducer.java:configure(79)) - child jvm opts: -Xmx200m
2015-02-12 16:12:51,031 INFO  hadoop.FlowReducer (FlowReducer.java:configure(97)) - sourcing from: GroupBy(first)[by:[{1}:'char']]
2015-02-12 16:12:51,032 INFO  hadoop.FlowReducer (FlowReducer.java:configure(100)) - sinking to: TempHfs["SequenceFile[['char', 'count']]"][6142022995/first/]
2015-02-12 16:12:51,040 INFO  mapred.Task (Task.java:done(1001)) - Task:attempt_local693942087_0020_r_000000_0 is done. And is in the process of committing
2015-02-12 16:12:51,041 INFO  mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) - 1 / 1 copied.
2015-02-12 16:12:51,041 INFO  mapred.Task (Task.java:commit(1162)) - Task attempt_local693942087_0020_r_000000_0 is allowed to commit now
2015-02-12 16:12:51,041 INFO  output.FileOutputCommitter (FileOutputCommitter.java:commitTask(439)) - Saved output of task 'attempt_local693942087_0020_r_000000_0' to file:/tmp/hadoop-soberoi/6142022995_first_F3B3041246EE453FA76367F7FF4B7C5C/_temporary/0/task_local693942087_0020_r_000000
2015-02-12 16:12:51,042 INFO  mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) - reduce > reduce
2015-02-12 16:12:51,042 INFO  mapred.Task (Task.java:sendDone(1121)) - Task 'attempt_local693942087_0020_r_000000_0' done.
2015-02-12 16:12:51,042 INFO  mapred.LocalJobRunner (LocalJobRunner.java:run(325)) - Finishing task: attempt_local693942087_0020_r_000000_0
2015-02-12 16:12:51,043 INFO  mapred.LocalJobRunner (LocalJobRunner.java:runTasks(456)) - reduce task executor complete.
2015-02-12 16:12:51,059 INFO  flow.FlowStep (BaseFlowStep.java:logInfo(834)) - [second] starting step: (2/2) ...form/samesourcemergecount
2015-02-12 16:12:51,060 INFO  jvm.JvmMetrics (JvmMetrics.java:init(71)) - Cannot initialize JVM Metrics with processName=JobTracker, sessionId= - already initialized
2015-02-12 16:12:51,061 INFO  jvm.JvmMetrics (JvmMetrics.java:init(71)) - Cannot initialize JVM Metrics with processName=JobTracker, sessionId= - already initialized
2015-02-12 16:12:51,093 INFO  mapred.FileInputFormat (FileInputFormat.java:listStatus(247)) - Total input paths to process : 1
2015-02-12 16:12:51,110 INFO  mapreduce.JobSubmitter (JobSubmitter.java:submitJobInternal(492)) - number of splits:1
2015-02-12 16:12:51,118 INFO  Configuration.deprecation (Configuration.java:warnOnceIfDeprecated(1022)) - mapred.used.genericoptionsparser is deprecated. Instead, use mapreduce.client.genericoptionsparser.used
2015-02-12 16:12:51,118 INFO  Configuration.deprecation (Configuration.java:warnOnceIfDeprecated(1022)) - mapred.job.tracker is deprecated. Instead, use mapreduce.jobtracker.address
2015-02-12 16:12:51,121 INFO  mapreduce.JobSubmitter (JobSubmitter.java:printTokens(575)) - Submitting tokens for job: job_local782660763_0021
2015-02-12 16:12:51,131 WARN  conf.Configuration (Configuration.java:loadProperty(2510)) - file:/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/tmp/cascading/staging/soberoi782660763/.staging/job_local782660763_0021/job.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.retry.interval;  Ignoring.
2015-02-12 16:12:51,132 WARN  conf.Configuration (Configuration.java:loadProperty(2510)) - file:/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/tmp/cascading/staging/soberoi782660763/.staging/job_local782660763_0021/job.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.attempts;  Ignoring.
2015-02-12 16:12:51,160 WARN  conf.Configuration (Configuration.java:loadProperty(2510)) - file:/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/test/tmp/mapred/local/localRunner/soberoi/job_local782660763_0021/job_local782660763_0021.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.retry.interval;  Ignoring.
2015-02-12 16:12:51,161 WARN  conf.Configuration (Configuration.java:loadProperty(2510)) - file:/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/test/tmp/mapred/local/localRunner/soberoi/job_local782660763_0021/job_local782660763_0021.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.attempts;  Ignoring.
2015-02-12 16:12:51,161 INFO  mapreduce.Job (Job.java:submit(1299)) - The url to track the job: http://localhost:8080/
2015-02-12 16:12:51,162 INFO  mapred.LocalJobRunner (LocalJobRunner.java:createOutputCommitter(471)) - OutputCommitter set in config null
2015-02-12 16:12:51,162 INFO  flow.FlowStep (BaseFlowStep.java:logInfo(834)) - [second] submitted hadoop job: job_local782660763_0021
2015-02-12 16:12:51,162 INFO  mapred.LocalJobRunner (LocalJobRunner.java:createOutputCommitter(489)) - OutputCommitter is org.apache.hadoop.mapred.FileOutputCommitter
2015-02-12 16:12:51,162 INFO  flow.FlowStep (BaseFlowStep.java:logInfo(834)) - [second] tracking url: http://localhost:8080/
2015-02-12 16:12:51,163 INFO  mapred.LocalJobRunner (LocalJobRunner.java:runTasks(448)) - Waiting for map tasks
2015-02-12 16:12:51,163 INFO  mapred.LocalJobRunner (LocalJobRunner.java:run(224)) - Starting task: attempt_local782660763_0021_m_000000_0
2015-02-12 16:12:51,164 INFO  util.ProcfsBasedProcessTree (ProcfsBasedProcessTree.java:isAvailable(181)) - ProcfsBasedProcessTree currently is supported only on Linux.
2015-02-12 16:12:51,164 INFO  mapred.Task (Task.java:initialize(587)) -  Using ResourceCalculatorProcessTree : null
2015-02-12 16:12:51,165 INFO  io.MultiInputSplit (MultiInputSplit.java:readFields(161)) - current split input path: file:/tmp/hadoop-soberoi/6142022995_first_F3B3041246EE453FA76367F7FF4B7C5C/part-00000
2015-02-12 16:12:51,165 INFO  mapred.MapTask (MapTask.java:updateJobWithSplit(482)) - Processing split: cascading.tap.hadoop.io.MultiInputSplit@6e9e36f1
2015-02-12 16:12:51,166 INFO  mapred.MapTask (MapTask.java:runOldMapper(436)) - numReduceTasks: 1
2015-02-12 16:12:51,181 INFO  mapred.MapTask (MapTask.java:setEquator(1202)) - (EQUATOR) 0 kvi 26214396(104857584)
2015-02-12 16:12:51,181 INFO  mapred.MapTask (MapTask.java:init(995)) - mapreduce.task.io.sort.mb: 100
2015-02-12 16:12:51,181 INFO  mapred.MapTask (MapTask.java:init(996)) - soft limit at 83886080
2015-02-12 16:12:51,181 INFO  mapred.MapTask (MapTask.java:init(997)) - bufstart = 0; bufvoid = 104857600
2015-02-12 16:12:51,181 INFO  mapred.MapTask (MapTask.java:init(998)) - kvstart = 26214396; length = 6553600
2015-02-12 16:12:51,182 INFO  mapred.MapTask (MapTask.java:createSortingCollector(402)) - Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer
2015-02-12 16:12:51,182 INFO  hadoop.FlowMapper (FlowMapper.java:configure(71)) - cascading version: 2.6.3
2015-02-12 16:12:51,182 INFO  hadoop.FlowMapper (FlowMapper.java:configure(72)) - child jvm opts: -Xmx200m
2015-02-12 16:12:51,187 INFO  hadoop.FlowMapper (FlowMapper.java:configure(87)) - sourcing from: TempHfs["SequenceFile[['char', 'count']]"][6142022995/first/]
2015-02-12 16:12:51,187 INFO  hadoop.FlowMapper (FlowMapper.java:configure(90)) - sinking to: GroupBy(second)[by:[{1}:'char']]
2015-02-12 16:12:51,188 INFO  assembly.AggregateBy (AggregateBy.java:prepare(267)) - using threshold value: 2
2015-02-12 16:12:51,189 INFO  assembly.AggregateBy (AggregateBy.java:removeEldestEntry(327)) - flushed keys num times: 1, with threshold: 2
2015-02-12 16:12:51,189 INFO  assembly.AggregateBy (AggregateBy.java:removeEldestEntry(328)) - mem on flush (mb), free: 196, total: 510, max: 672
2015-02-12 16:12:51,189 INFO  assembly.AggregateBy (AggregateBy.java:removeEldestEntry(333)) - total mem is 75% of max mem, to better utilize unused memory consider increasing current LRU threshold with system property "cascading.aggregateby.threshold"
2015-02-12 16:12:51,189 INFO  assembly.AggregateBy (AggregateBy.java:removeEldestEntry(327)) - flushed keys num times: 3, with threshold: 2
2015-02-12 16:12:51,189 INFO  assembly.AggregateBy (AggregateBy.java:removeEldestEntry(328)) - mem on flush (mb), free: 196, total: 510, max: 672
2015-02-12 16:12:51,189 INFO  assembly.AggregateBy (AggregateBy.java:removeEldestEntry(333)) - total mem is 75% of max mem, to better utilize unused memory consider increasing current LRU threshold with system property "cascading.aggregateby.threshold"
2015-02-12 16:12:51,191 INFO  mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) - 
2015-02-12 16:12:51,191 INFO  mapred.MapTask (MapTask.java:flush(1457)) - Starting flush of map output
2015-02-12 16:12:51,191 INFO  mapred.MapTask (MapTask.java:flush(1475)) - Spilling map output
2015-02-12 16:12:51,191 INFO  mapred.MapTask (MapTask.java:flush(1476)) - bufstart = 0; bufend = 55; bufvoid = 104857600
2015-02-12 16:12:51,191 INFO  mapred.MapTask (MapTask.java:flush(1478)) - kvstart = 26214396(104857584); kvend = 26214380(104857520); length = 17/6553600
2015-02-12 16:12:51,193 INFO  mapred.MapTask (MapTask.java:sortAndSpill(1660)) - Finished spill 0
2015-02-12 16:12:51,194 INFO  mapred.Task (Task.java:done(1001)) - Task:attempt_local782660763_0021_m_000000_0 is done. And is in the process of committing
2015-02-12 16:12:51,195 INFO  mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) - file:/tmp/hadoop-soberoi/6142022995_first_F3B3041246EE453FA76367F7FF4B7C5C/part-00000:0+160
2015-02-12 16:12:51,195 INFO  mapred.Task (Task.java:sendDone(1121)) - Task 'attempt_local782660763_0021_m_000000_0' done.
2015-02-12 16:12:51,195 INFO  mapred.LocalJobRunner (LocalJobRunner.java:run(249)) - Finishing task: attempt_local782660763_0021_m_000000_0
2015-02-12 16:12:51,195 INFO  mapred.LocalJobRunner (LocalJobRunner.java:runTasks(456)) - map task executor complete.
2015-02-12 16:12:51,195 INFO  mapred.LocalJobRunner (LocalJobRunner.java:runTasks(448)) - Waiting for reduce tasks
2015-02-12 16:12:51,195 INFO  mapred.LocalJobRunner (LocalJobRunner.java:run(302)) - Starting task: attempt_local782660763_0021_r_000000_0
2015-02-12 16:12:51,196 INFO  util.ProcfsBasedProcessTree (ProcfsBasedProcessTree.java:isAvailable(181)) - ProcfsBasedProcessTree currently is supported only on Linux.
2015-02-12 16:12:51,197 INFO  mapred.Task (Task.java:initialize(587)) -  Using ResourceCalculatorProcessTree : null
2015-02-12 16:12:51,197 INFO  mapred.ReduceTask (ReduceTask.java:run(362)) - Using ShuffleConsumerPlugin: org.apache.hadoop.mapreduce.task.reduce.Shuffle@67ea92f3
2015-02-12 16:12:51,197 INFO  reduce.MergeManagerImpl (MergeManagerImpl.java:<init>(195)) - MergerManager: memoryLimit=493250144, maxSingleShuffleLimit=123312536, mergeThreshold=325545120, ioSortFactor=10, memToMemMergeOutputsThreshold=10
2015-02-12 16:12:51,197 INFO  reduce.EventFetcher (EventFetcher.java:run(61)) - attempt_local782660763_0021_r_000000_0 Thread started: EventFetcher for fetching Map Completion Events
2015-02-12 16:12:51,198 INFO  reduce.LocalFetcher (LocalFetcher.java:copyMapOutput(141)) - localfetcher#17 about to shuffle output of map attempt_local782660763_0021_m_000000_0 decomp: 67 len: 71 to MEMORY
2015-02-12 16:12:51,198 INFO  reduce.InMemoryMapOutput (InMemoryMapOutput.java:shuffle(100)) - Read 67 bytes from map-output for attempt_local782660763_0021_m_000000_0
2015-02-12 16:12:51,199 INFO  reduce.MergeManagerImpl (MergeManagerImpl.java:closeInMemoryFile(313)) - closeInMemoryFile -> map-output of size: 67, inMemoryMapOutputs.size() -> 1, commitMemory -> 0, usedMemory ->67
2015-02-12 16:12:51,199 INFO  reduce.EventFetcher (EventFetcher.java:run(76)) - EventFetcher is interrupted.. Returning
2015-02-12 16:12:51,199 INFO  mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) - 1 / 1 copied.
2015-02-12 16:12:51,200 INFO  reduce.MergeManagerImpl (MergeManagerImpl.java:finalMerge(673)) - finalMerge called with 1 in-memory map-outputs and 0 on-disk map-outputs
2015-02-12 16:12:51,201 INFO  mapred.Merger (Merger.java:merge(606)) - Merging 1 sorted segments
2015-02-12 16:12:51,201 INFO  mapred.Merger (Merger.java:merge(705)) - Down to the last merge-pass, with 1 segments left of total size: 58 bytes
2015-02-12 16:12:51,202 INFO  reduce.MergeManagerImpl (MergeManagerImpl.java:finalMerge(750)) - Merged 1 segments, 67 bytes to disk to satisfy reduce memory limit
2015-02-12 16:12:51,202 INFO  reduce.MergeManagerImpl (MergeManagerImpl.java:finalMerge(780)) - Merging 1 files, 71 bytes from disk
2015-02-12 16:12:51,202 INFO  reduce.MergeManagerImpl (MergeManagerImpl.java:finalMerge(795)) - Merging 0 segments, 0 bytes from memory into reduce
2015-02-12 16:12:51,202 INFO  mapred.Merger (Merger.java:merge(606)) - Merging 1 sorted segments
2015-02-12 16:12:51,202 INFO  mapred.Merger (Merger.java:merge(705)) - Down to the last merge-pass, with 1 segments left of total size: 58 bytes
2015-02-12 16:12:51,203 INFO  mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) - 1 / 1 copied.
2015-02-12 16:12:51,203 INFO  hadoop.FlowReducer (FlowReducer.java:configure(78)) - cascading version: 2.6.3
2015-02-12 16:12:51,203 INFO  hadoop.FlowReducer (FlowReducer.java:configure(79)) - child jvm opts: -Xmx200m
2015-02-12 16:12:51,209 INFO  hadoop.FlowReducer (FlowReducer.java:configure(97)) - sourcing from: GroupBy(second)[by:[{1}:'char']]
2015-02-12 16:12:51,209 INFO  hadoop.FlowReducer (FlowReducer.java:configure(100)) - sinking to: Hfs["TextDelimited[['char', 'count']]"]["/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/test/output/hadoop2-mr1/assemblyhelpersplatform/samesourcemergecount"]
2015-02-12 16:12:51,217 INFO  mapred.Task (Task.java:done(1001)) - Task:attempt_local782660763_0021_r_000000_0 is done. And is in the process of committing
2015-02-12 16:12:51,218 INFO  mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) - 1 / 1 copied.
2015-02-12 16:12:51,218 INFO  mapred.Task (Task.java:commit(1162)) - Task attempt_local782660763_0021_r_000000_0 is allowed to commit now
2015-02-12 16:12:51,219 INFO  output.FileOutputCommitter (FileOutputCommitter.java:commitTask(439)) - Saved output of task 'attempt_local782660763_0021_r_000000_0' to file:/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/test/output/hadoop2-mr1/assemblyhelpersplatform/samesourcemergecount/_temporary/0/task_local782660763_0021_r_000000
2015-02-12 16:12:51,219 INFO  mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) - reduce > reduce
2015-02-12 16:12:51,219 INFO  mapred.Task (Task.java:sendDone(1121)) - Task 'attempt_local782660763_0021_r_000000_0' done.
2015-02-12 16:12:51,220 INFO  mapred.LocalJobRunner (LocalJobRunner.java:run(325)) - Finishing task: attempt_local782660763_0021_r_000000_0
2015-02-12 16:12:51,220 INFO  mapred.LocalJobRunner (LocalJobRunner.java:runTasks(456)) - reduce task executor complete.
2015-02-12 16:12:51,238 INFO  util.Hadoop18TapUtil (Hadoop18TapUtil.java:cleanTempPath(219)) - deleting temp path /Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/test/output/hadoop2-mr1/assemblyhelpersplatform/samesourcemergecount/_temporary
2015-02-12 16:12:51,257 INFO  mapred.FileInputFormat (FileInputFormat.java:listStatus(247)) - Total input paths to process : 1
2015-02-12 16:12:51,278 INFO  mapred.FileInputFormat (FileInputFormat.java:listStatus(247)) - Total input paths to process : 1
2015-02-12 16:12:51,279 INFO  cascading.PlatformTestCase (PlatformTestCase.java:tearDown(209)) - copying to local /Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/test/output//hadoop2-mr1/assemblyhelpersplatform/samesourcemergecount
2015-02-12 16:12:51,290 INFO  util.HadoopUtil (HadoopUtil.java:findMainClass(336)) - using default application jar, may cause class not found exceptions on the cluster
2015-02-12 16:12:51,290 INFO  planner.HadoopPlanner (HadoopPlanner.java:initialize(225)) - using application jar: /Users/soberoi/.gradle/caches/modules-2/files-2.1/cascading/cascading-hadoop2-mr1/2.6.3/9ae016e2b8d246b9c1de9168e715d9f1b971aa4a/cascading-hadoop2-mr1-2.6.3.jar
2015-02-12 16:12:51,293 INFO  Configuration.deprecation (Configuration.java:warnOnceIfDeprecated(1022)) - mapred.used.genericoptionsparser is deprecated. Instead, use mapreduce.client.genericoptionsparser.used
2015-02-12 16:12:51,294 INFO  Configuration.deprecation (Configuration.java:warnOnceIfDeprecated(1022)) - mapred.job.tracker is deprecated. Instead, use mapreduce.jobtracker.address
2015-02-12 16:12:51,294 INFO  Configuration.deprecation (Configuration.java:warnOnceIfDeprecated(1022)) - mapred.output.compress is deprecated. Instead, use mapreduce.output.fileoutputformat.compress
2015-02-12 16:12:51,298 INFO  flow.Flow (BaseFlow.java:logInfo(1378)) - [average] starting
2015-02-12 16:12:51,298 INFO  flow.Flow (BaseFlow.java:logInfo(1378)) - [average]  source: Hfs["TextDelimited[['num', 'char']]"]["/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/classes/test/data/lhs.txt"]
2015-02-12 16:12:51,298 INFO  flow.Flow (BaseFlow.java:logInfo(1378)) - [average]  sink: Hfs["TextDelimited[['char', 'average']]"]["/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/test/output/hadoop2-mr1/assemblyhelpersplatform/average"]
2015-02-12 16:12:51,298 INFO  flow.Flow (BaseFlow.java:logInfo(1378)) - [average]  parallel execution is enabled: false
2015-02-12 16:12:51,298 INFO  flow.Flow (BaseFlow.java:logInfo(1378)) - [average]  starting jobs: 1
2015-02-12 16:12:51,298 INFO  flow.Flow (BaseFlow.java:logInfo(1378)) - [average]  allocating threads: 1
2015-02-12 16:12:51,299 INFO  flow.FlowStep (BaseFlowStep.java:logInfo(834)) - [average] starting step: (1/1) ...lyhelpersplatform/average
2015-02-12 16:12:51,299 INFO  jvm.JvmMetrics (JvmMetrics.java:init(71)) - Cannot initialize JVM Metrics with processName=JobTracker, sessionId= - already initialized
2015-02-12 16:12:51,300 INFO  jvm.JvmMetrics (JvmMetrics.java:init(71)) - Cannot initialize JVM Metrics with processName=JobTracker, sessionId= - already initialized
2015-02-12 16:12:51,319 INFO  mapred.FileInputFormat (FileInputFormat.java:listStatus(247)) - Total input paths to process : 1
2015-02-12 16:12:51,334 INFO  mapreduce.JobSubmitter (JobSubmitter.java:submitJobInternal(492)) - number of splits:1
2015-02-12 16:12:51,345 INFO  mapreduce.JobSubmitter (JobSubmitter.java:printTokens(575)) - Submitting tokens for job: job_local1315145036_0022
2015-02-12 16:12:51,358 WARN  conf.Configuration (Configuration.java:loadProperty(2510)) - file:/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/tmp/cascading/staging/soberoi1315145036/.staging/job_local1315145036_0022/job.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.retry.interval;  Ignoring.
2015-02-12 16:12:51,359 WARN  conf.Configuration (Configuration.java:loadProperty(2510)) - file:/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/tmp/cascading/staging/soberoi1315145036/.staging/job_local1315145036_0022/job.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.attempts;  Ignoring.
2015-02-12 16:12:51,389 WARN  conf.Configuration (Configuration.java:loadProperty(2510)) - file:/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/test/tmp/mapred/local/localRunner/soberoi/job_local1315145036_0022/job_local1315145036_0022.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.retry.interval;  Ignoring.
2015-02-12 16:12:51,390 WARN  conf.Configuration (Configuration.java:loadProperty(2510)) - file:/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/test/tmp/mapred/local/localRunner/soberoi/job_local1315145036_0022/job_local1315145036_0022.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.attempts;  Ignoring.
2015-02-12 16:12:51,391 INFO  mapreduce.Job (Job.java:submit(1299)) - The url to track the job: http://localhost:8080/
2015-02-12 16:12:51,391 INFO  flow.FlowStep (BaseFlowStep.java:logInfo(834)) - [average] submitted hadoop job: job_local1315145036_0022
2015-02-12 16:12:51,391 INFO  mapred.LocalJobRunner (LocalJobRunner.java:createOutputCommitter(471)) - OutputCommitter set in config null
2015-02-12 16:12:51,391 INFO  flow.FlowStep (BaseFlowStep.java:logInfo(834)) - [average] tracking url: http://localhost:8080/
2015-02-12 16:12:51,391 INFO  mapred.LocalJobRunner (LocalJobRunner.java:createOutputCommitter(489)) - OutputCommitter is org.apache.hadoop.mapred.FileOutputCommitter
2015-02-12 16:12:51,393 INFO  mapred.LocalJobRunner (LocalJobRunner.java:runTasks(448)) - Waiting for map tasks
2015-02-12 16:12:51,393 INFO  mapred.LocalJobRunner (LocalJobRunner.java:run(224)) - Starting task: attempt_local1315145036_0022_m_000000_0
2015-02-12 16:12:51,394 INFO  util.ProcfsBasedProcessTree (ProcfsBasedProcessTree.java:isAvailable(181)) - ProcfsBasedProcessTree currently is supported only on Linux.
2015-02-12 16:12:51,394 INFO  mapred.Task (Task.java:initialize(587)) -  Using ResourceCalculatorProcessTree : null
2015-02-12 16:12:51,394 INFO  io.MultiInputSplit (MultiInputSplit.java:readFields(161)) - current split input path: file:/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/classes/test/data/lhs.txt
2015-02-12 16:12:51,395 INFO  mapred.MapTask (MapTask.java:updateJobWithSplit(482)) - Processing split: cascading.tap.hadoop.io.MultiInputSplit@41b2fd97
2015-02-12 16:12:51,395 INFO  mapred.MapTask (MapTask.java:runOldMapper(436)) - numReduceTasks: 1
2015-02-12 16:12:51,410 INFO  mapred.MapTask (MapTask.java:setEquator(1202)) - (EQUATOR) 0 kvi 26214396(104857584)
2015-02-12 16:12:51,410 INFO  mapred.MapTask (MapTask.java:init(995)) - mapreduce.task.io.sort.mb: 100
2015-02-12 16:12:51,410 INFO  mapred.MapTask (MapTask.java:init(996)) - soft limit at 83886080
2015-02-12 16:12:51,410 INFO  mapred.MapTask (MapTask.java:init(997)) - bufstart = 0; bufvoid = 104857600
2015-02-12 16:12:51,410 INFO  mapred.MapTask (MapTask.java:init(998)) - kvstart = 26214396; length = 6553600
2015-02-12 16:12:51,411 INFO  mapred.MapTask (MapTask.java:createSortingCollector(402)) - Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer
2015-02-12 16:12:51,411 INFO  hadoop.FlowMapper (FlowMapper.java:configure(71)) - cascading version: 2.6.3
2015-02-12 16:12:51,411 INFO  hadoop.FlowMapper (FlowMapper.java:configure(72)) - child jvm opts: -Xmx200m
2015-02-12 16:12:51,415 INFO  hadoop.FlowMapper (FlowMapper.java:configure(87)) - sourcing from: Hfs["TextDelimited[['num', 'char']]"]["/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/classes/test/data/lhs.txt"]
2015-02-12 16:12:51,415 INFO  hadoop.FlowMapper (FlowMapper.java:configure(90)) - sinking to: GroupBy(average)[by:[{1}:'char']]
2015-02-12 16:12:51,415 INFO  assembly.AggregateBy (AggregateBy.java:prepare(267)) - using threshold value: 2
2015-02-12 16:12:51,416 INFO  assembly.AggregateBy (AggregateBy.java:removeEldestEntry(327)) - flushed keys num times: 1, with threshold: 2
2015-02-12 16:12:51,416 INFO  assembly.AggregateBy (AggregateBy.java:removeEldestEntry(328)) - mem on flush (mb), free: 197, total: 511, max: 672
2015-02-12 16:12:51,416 INFO  assembly.AggregateBy (AggregateBy.java:removeEldestEntry(333)) - total mem is 76% of max mem, to better utilize unused memory consider increasing current LRU threshold with system property "cascading.aggregateby.threshold"
2015-02-12 16:12:51,417 INFO  assembly.AggregateBy (AggregateBy.java:removeEldestEntry(327)) - flushed keys num times: 3, with threshold: 2
2015-02-12 16:12:51,417 INFO  assembly.AggregateBy (AggregateBy.java:removeEldestEntry(328)) - mem on flush (mb), free: 197, total: 511, max: 672
2015-02-12 16:12:51,417 INFO  assembly.AggregateBy (AggregateBy.java:removeEldestEntry(333)) - total mem is 76% of max mem, to better utilize unused memory consider increasing current LRU threshold with system property "cascading.aggregateby.threshold"
2015-02-12 16:12:51,417 INFO  assembly.AggregateBy (AggregateBy.java:removeEldestEntry(327)) - flushed keys num times: 5, with threshold: 2
2015-02-12 16:12:51,418 INFO  assembly.AggregateBy (AggregateBy.java:removeEldestEntry(328)) - mem on flush (mb), free: 196, total: 511, max: 672
2015-02-12 16:12:51,418 INFO  assembly.AggregateBy (AggregateBy.java:removeEldestEntry(333)) - total mem is 76% of max mem, to better utilize unused memory consider increasing current LRU threshold with system property "cascading.aggregateby.threshold"
2015-02-12 16:12:51,418 INFO  assembly.AggregateBy (AggregateBy.java:removeEldestEntry(327)) - flushed keys num times: 7, with threshold: 2
2015-02-12 16:12:51,418 INFO  assembly.AggregateBy (AggregateBy.java:removeEldestEntry(328)) - mem on flush (mb), free: 196, total: 511, max: 672
2015-02-12 16:12:51,418 INFO  assembly.AggregateBy (AggregateBy.java:removeEldestEntry(333)) - total mem is 76% of max mem, to better utilize unused memory consider increasing current LRU threshold with system property "cascading.aggregateby.threshold"
2015-02-12 16:12:51,419 INFO  mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) - 
2015-02-12 16:12:51,420 INFO  mapred.MapTask (MapTask.java:flush(1457)) - Starting flush of map output
2015-02-12 16:12:51,420 INFO  mapred.MapTask (MapTask.java:flush(1475)) - Spilling map output
2015-02-12 16:12:51,420 INFO  mapred.MapTask (MapTask.java:flush(1476)) - bufstart = 0; bufend = 180; bufvoid = 104857600
2015-02-12 16:12:51,420 INFO  mapred.MapTask (MapTask.java:flush(1478)) - kvstart = 26214396(104857584); kvend = 26214364(104857456); length = 33/6553600
2015-02-12 16:12:51,422 INFO  mapred.MapTask (MapTask.java:sortAndSpill(1660)) - Finished spill 0
2015-02-12 16:12:51,423 INFO  mapred.Task (Task.java:done(1001)) - Task:attempt_local1315145036_0022_m_000000_0 is done. And is in the process of committing
2015-02-12 16:12:51,424 INFO  mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) - file:/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/classes/test/data/lhs.txt:0+52
2015-02-12 16:12:51,424 INFO  mapred.Task (Task.java:sendDone(1121)) - Task 'attempt_local1315145036_0022_m_000000_0' done.
2015-02-12 16:12:51,424 INFO  mapred.LocalJobRunner (LocalJobRunner.java:run(249)) - Finishing task: attempt_local1315145036_0022_m_000000_0
2015-02-12 16:12:51,424 INFO  mapred.LocalJobRunner (LocalJobRunner.java:runTasks(456)) - map task executor complete.
2015-02-12 16:12:51,425 INFO  mapred.LocalJobRunner (LocalJobRunner.java:runTasks(448)) - Waiting for reduce tasks
2015-02-12 16:12:51,425 INFO  mapred.LocalJobRunner (LocalJobRunner.java:run(302)) - Starting task: attempt_local1315145036_0022_r_000000_0
2015-02-12 16:12:51,426 INFO  util.ProcfsBasedProcessTree (ProcfsBasedProcessTree.java:isAvailable(181)) - ProcfsBasedProcessTree currently is supported only on Linux.
2015-02-12 16:12:51,426 INFO  mapred.Task (Task.java:initialize(587)) -  Using ResourceCalculatorProcessTree : null
2015-02-12 16:12:51,426 INFO  mapred.ReduceTask (ReduceTask.java:run(362)) - Using ShuffleConsumerPlugin: org.apache.hadoop.mapreduce.task.reduce.Shuffle@7f317c99
2015-02-12 16:12:51,427 INFO  reduce.MergeManagerImpl (MergeManagerImpl.java:<init>(195)) - MergerManager: memoryLimit=493250144, maxSingleShuffleLimit=123312536, mergeThreshold=325545120, ioSortFactor=10, memToMemMergeOutputsThreshold=10
2015-02-12 16:12:51,428 INFO  reduce.EventFetcher (EventFetcher.java:run(61)) - attempt_local1315145036_0022_r_000000_0 Thread started: EventFetcher for fetching Map Completion Events
2015-02-12 16:12:51,428 INFO  reduce.LocalFetcher (LocalFetcher.java:copyMapOutput(141)) - localfetcher#18 about to shuffle output of map attempt_local1315145036_0022_m_000000_0 decomp: 200 len: 204 to MEMORY
2015-02-12 16:12:51,429 INFO  reduce.InMemoryMapOutput (InMemoryMapOutput.java:shuffle(100)) - Read 200 bytes from map-output for attempt_local1315145036_0022_m_000000_0
2015-02-12 16:12:51,429 INFO  reduce.MergeManagerImpl (MergeManagerImpl.java:closeInMemoryFile(313)) - closeInMemoryFile -> map-output of size: 200, inMemoryMapOutputs.size() -> 1, commitMemory -> 0, usedMemory ->200
2015-02-12 16:12:51,429 INFO  reduce.EventFetcher (EventFetcher.java:run(76)) - EventFetcher is interrupted.. Returning
2015-02-12 16:12:51,430 INFO  mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) - 1 / 1 copied.
2015-02-12 16:12:51,430 INFO  reduce.MergeManagerImpl (MergeManagerImpl.java:finalMerge(673)) - finalMerge called with 1 in-memory map-outputs and 0 on-disk map-outputs
2015-02-12 16:12:51,431 INFO  mapred.Merger (Merger.java:merge(606)) - Merging 1 sorted segments
2015-02-12 16:12:51,431 INFO  mapred.Merger (Merger.java:merge(705)) - Down to the last merge-pass, with 1 segments left of total size: 191 bytes
2015-02-12 16:12:51,432 INFO  reduce.MergeManagerImpl (MergeManagerImpl.java:finalMerge(750)) - Merged 1 segments, 200 bytes to disk to satisfy reduce memory limit
2015-02-12 16:12:51,433 INFO  reduce.MergeManagerImpl (MergeManagerImpl.java:finalMerge(780)) - Merging 1 files, 204 bytes from disk
2015-02-12 16:12:51,433 INFO  reduce.MergeManagerImpl (MergeManagerImpl.java:finalMerge(795)) - Merging 0 segments, 0 bytes from memory into reduce
2015-02-12 16:12:51,433 INFO  mapred.Merger (Merger.java:merge(606)) - Merging 1 sorted segments
2015-02-12 16:12:51,433 INFO  mapred.Merger (Merger.java:merge(705)) - Down to the last merge-pass, with 1 segments left of total size: 191 bytes
2015-02-12 16:12:51,433 INFO  mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) - 1 / 1 copied.
2015-02-12 16:12:51,434 INFO  hadoop.FlowReducer (FlowReducer.java:configure(78)) - cascading version: 2.6.3
2015-02-12 16:12:51,434 INFO  hadoop.FlowReducer (FlowReducer.java:configure(79)) - child jvm opts: -Xmx200m
2015-02-12 16:12:51,438 INFO  hadoop.FlowReducer (FlowReducer.java:configure(97)) - sourcing from: GroupBy(average)[by:[{1}:'char']]
2015-02-12 16:12:51,439 INFO  hadoop.FlowReducer (FlowReducer.java:configure(100)) - sinking to: Hfs["TextDelimited[['char', 'average']]"]["/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/test/output/hadoop2-mr1/assemblyhelpersplatform/average"]
2015-02-12 16:12:51,447 INFO  mapred.Task (Task.java:done(1001)) - Task:attempt_local1315145036_0022_r_000000_0 is done. And is in the process of committing
2015-02-12 16:12:51,448 INFO  mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) - 1 / 1 copied.
2015-02-12 16:12:51,448 INFO  mapred.Task (Task.java:commit(1162)) - Task attempt_local1315145036_0022_r_000000_0 is allowed to commit now
2015-02-12 16:12:51,449 INFO  output.FileOutputCommitter (FileOutputCommitter.java:commitTask(439)) - Saved output of task 'attempt_local1315145036_0022_r_000000_0' to file:/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/test/output/hadoop2-mr1/assemblyhelpersplatform/average/_temporary/0/task_local1315145036_0022_r_000000
2015-02-12 16:12:51,450 INFO  mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) - reduce > reduce
2015-02-12 16:12:51,450 INFO  mapred.Task (Task.java:sendDone(1121)) - Task 'attempt_local1315145036_0022_r_000000_0' done.
2015-02-12 16:12:51,450 INFO  mapred.LocalJobRunner (LocalJobRunner.java:run(325)) - Finishing task: attempt_local1315145036_0022_r_000000_0
2015-02-12 16:12:51,450 INFO  mapred.LocalJobRunner (LocalJobRunner.java:runTasks(456)) - reduce task executor complete.
2015-02-12 16:12:51,461 INFO  util.Hadoop18TapUtil (Hadoop18TapUtil.java:cleanTempPath(219)) - deleting temp path /Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/test/output/hadoop2-mr1/assemblyhelpersplatform/average/_temporary
2015-02-12 16:12:51,487 INFO  mapred.FileInputFormat (FileInputFormat.java:listStatus(247)) - Total input paths to process : 1
2015-02-12 16:12:51,507 INFO  mapred.FileInputFormat (FileInputFormat.java:listStatus(247)) - Total input paths to process : 1
2015-02-12 16:12:51,508 INFO  cascading.PlatformTestCase (PlatformTestCase.java:tearDown(209)) - copying to local /Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/test/output//hadoop2-mr1/assemblyhelpersplatform/average
2015-02-12 16:12:51,518 INFO  util.HadoopUtil (HadoopUtil.java:findMainClass(336)) - using default application jar, may cause class not found exceptions on the cluster
2015-02-12 16:12:51,518 INFO  planner.HadoopPlanner (HadoopPlanner.java:initialize(225)) - using application jar: /Users/soberoi/.gradle/caches/modules-2/files-2.1/cascading/cascading-hadoop2-mr1/2.6.3/9ae016e2b8d246b9c1de9168e715d9f1b971aa4a/cascading-hadoop2-mr1-2.6.3.jar
2015-02-12 16:12:51,522 INFO  Configuration.deprecation (Configuration.java:warnOnceIfDeprecated(1022)) - mapred.used.genericoptionsparser is deprecated. Instead, use mapreduce.client.genericoptionsparser.used
2015-02-12 16:12:51,523 INFO  Configuration.deprecation (Configuration.java:warnOnceIfDeprecated(1022)) - mapred.job.tracker is deprecated. Instead, use mapreduce.jobtracker.address
2015-02-12 16:12:51,524 INFO  Configuration.deprecation (Configuration.java:warnOnceIfDeprecated(1022)) - mapred.output.compress is deprecated. Instead, use mapreduce.output.fileoutputformat.compress
2015-02-12 16:12:51,528 INFO  flow.Flow (BaseFlow.java:logInfo(1378)) - [max] starting
2015-02-12 16:12:51,528 INFO  flow.Flow (BaseFlow.java:logInfo(1378)) - [max]  source: Hfs["TextDelimited[['num', 'char']]"]["/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/classes/test/data/lhs.txt"]
2015-02-12 16:12:51,528 INFO  flow.Flow (BaseFlow.java:logInfo(1378)) - [max]  sink: Hfs["TextDelimited[['char', 'max']]"]["/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/test/output/hadoop2-mr1/assemblyhelpersplatform/maxbynullsafety"]
2015-02-12 16:12:51,529 INFO  flow.Flow (BaseFlow.java:logInfo(1378)) - [max]  parallel execution is enabled: false
2015-02-12 16:12:51,529 INFO  flow.Flow (BaseFlow.java:logInfo(1378)) - [max]  starting jobs: 1
2015-02-12 16:12:51,529 INFO  flow.Flow (BaseFlow.java:logInfo(1378)) - [max]  allocating threads: 1
2015-02-12 16:12:51,529 INFO  flow.FlowStep (BaseFlowStep.java:logInfo(834)) - [max] starting step: (1/1) ...splatform/maxbynullsafety
2015-02-12 16:12:51,530 INFO  jvm.JvmMetrics (JvmMetrics.java:init(71)) - Cannot initialize JVM Metrics with processName=JobTracker, sessionId= - already initialized
2015-02-12 16:12:51,530 INFO  jvm.JvmMetrics (JvmMetrics.java:init(71)) - Cannot initialize JVM Metrics with processName=JobTracker, sessionId= - already initialized
2015-02-12 16:12:51,547 INFO  mapred.FileInputFormat (FileInputFormat.java:listStatus(247)) - Total input paths to process : 1
2015-02-12 16:12:51,563 INFO  mapreduce.JobSubmitter (JobSubmitter.java:submitJobInternal(492)) - number of splits:1
2015-02-12 16:12:51,575 INFO  mapreduce.JobSubmitter (JobSubmitter.java:printTokens(575)) - Submitting tokens for job: job_local1388077210_0023
2015-02-12 16:12:51,587 WARN  conf.Configuration (Configuration.java:loadProperty(2510)) - file:/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/tmp/cascading/staging/soberoi1388077210/.staging/job_local1388077210_0023/job.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.retry.interval;  Ignoring.
2015-02-12 16:12:51,587 WARN  conf.Configuration (Configuration.java:loadProperty(2510)) - file:/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/tmp/cascading/staging/soberoi1388077210/.staging/job_local1388077210_0023/job.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.attempts;  Ignoring.
2015-02-12 16:12:51,614 WARN  conf.Configuration (Configuration.java:loadProperty(2510)) - file:/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/test/tmp/mapred/local/localRunner/soberoi/job_local1388077210_0023/job_local1388077210_0023.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.retry.interval;  Ignoring.
2015-02-12 16:12:51,615 WARN  conf.Configuration (Configuration.java:loadProperty(2510)) - file:/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/test/tmp/mapred/local/localRunner/soberoi/job_local1388077210_0023/job_local1388077210_0023.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.attempts;  Ignoring.
2015-02-12 16:12:51,615 INFO  mapreduce.Job (Job.java:submit(1299)) - The url to track the job: http://localhost:8080/
2015-02-12 16:12:51,615 INFO  mapred.LocalJobRunner (LocalJobRunner.java:createOutputCommitter(471)) - OutputCommitter set in config null
2015-02-12 16:12:51,615 INFO  flow.FlowStep (BaseFlowStep.java:logInfo(834)) - [max] submitted hadoop job: job_local1388077210_0023
2015-02-12 16:12:51,616 INFO  flow.FlowStep (BaseFlowStep.java:logInfo(834)) - [max] tracking url: http://localhost:8080/
2015-02-12 16:12:51,616 INFO  mapred.LocalJobRunner (LocalJobRunner.java:createOutputCommitter(489)) - OutputCommitter is org.apache.hadoop.mapred.FileOutputCommitter
2015-02-12 16:12:51,617 INFO  mapred.LocalJobRunner (LocalJobRunner.java:runTasks(448)) - Waiting for map tasks
2015-02-12 16:12:51,617 INFO  mapred.LocalJobRunner (LocalJobRunner.java:run(224)) - Starting task: attempt_local1388077210_0023_m_000000_0
2015-02-12 16:12:51,618 INFO  util.ProcfsBasedProcessTree (ProcfsBasedProcessTree.java:isAvailable(181)) - ProcfsBasedProcessTree currently is supported only on Linux.
2015-02-12 16:12:51,618 INFO  mapred.Task (Task.java:initialize(587)) -  Using ResourceCalculatorProcessTree : null
2015-02-12 16:12:51,619 INFO  io.MultiInputSplit (MultiInputSplit.java:readFields(161)) - current split input path: file:/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/classes/test/data/lhs.txt
2015-02-12 16:12:51,619 INFO  mapred.MapTask (MapTask.java:updateJobWithSplit(482)) - Processing split: cascading.tap.hadoop.io.MultiInputSplit@17ce3e76
2015-02-12 16:12:51,620 INFO  mapred.MapTask (MapTask.java:runOldMapper(436)) - numReduceTasks: 1
2015-02-12 16:12:51,636 INFO  mapred.MapTask (MapTask.java:setEquator(1202)) - (EQUATOR) 0 kvi 26214396(104857584)
2015-02-12 16:12:51,636 INFO  mapred.MapTask (MapTask.java:init(995)) - mapreduce.task.io.sort.mb: 100
2015-02-12 16:12:51,636 INFO  mapred.MapTask (MapTask.java:init(996)) - soft limit at 83886080
2015-02-12 16:12:51,636 INFO  mapred.MapTask (MapTask.java:init(997)) - bufstart = 0; bufvoid = 104857600
2015-02-12 16:12:51,636 INFO  mapred.MapTask (MapTask.java:init(998)) - kvstart = 26214396; length = 6553600
2015-02-12 16:12:51,637 INFO  mapred.MapTask (MapTask.java:createSortingCollector(402)) - Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer
2015-02-12 16:12:51,637 INFO  hadoop.FlowMapper (FlowMapper.java:configure(71)) - cascading version: 2.6.3
2015-02-12 16:12:51,637 INFO  hadoop.FlowMapper (FlowMapper.java:configure(72)) - child jvm opts: -Xmx200m
2015-02-12 16:12:51,640 INFO  hadoop.FlowMapper (FlowMapper.java:configure(87)) - sourcing from: Hfs["TextDelimited[['num', 'char']]"]["/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/classes/test/data/lhs.txt"]
2015-02-12 16:12:51,640 INFO  hadoop.FlowMapper (FlowMapper.java:configure(90)) - sinking to: GroupBy(max)[by:[{1}:'char']]
2015-02-12 16:12:51,641 INFO  assembly.AggregateBy (AggregateBy.java:prepare(267)) - using threshold value: 2
2015-02-12 16:12:51,641 INFO  assembly.AggregateBy (AggregateBy.java:removeEldestEntry(327)) - flushed keys num times: 1, with threshold: 2
2015-02-12 16:12:51,641 INFO  assembly.AggregateBy (AggregateBy.java:removeEldestEntry(328)) - mem on flush (mb), free: 196, total: 511, max: 672
2015-02-12 16:12:51,642 INFO  assembly.AggregateBy (AggregateBy.java:removeEldestEntry(333)) - total mem is 76% of max mem, to better utilize unused memory consider increasing current LRU threshold with system property "cascading.aggregateby.threshold"
2015-02-12 16:12:51,642 INFO  assembly.AggregateBy (AggregateBy.java:removeEldestEntry(327)) - flushed keys num times: 3, with threshold: 2
2015-02-12 16:12:51,642 INFO  assembly.AggregateBy (AggregateBy.java:removeEldestEntry(328)) - mem on flush (mb), free: 196, total: 511, max: 672
2015-02-12 16:12:51,642 INFO  assembly.AggregateBy (AggregateBy.java:removeEldestEntry(333)) - total mem is 76% of max mem, to better utilize unused memory consider increasing current LRU threshold with system property "cascading.aggregateby.threshold"
2015-02-12 16:12:51,643 INFO  assembly.AggregateBy (AggregateBy.java:removeEldestEntry(327)) - flushed keys num times: 5, with threshold: 2
2015-02-12 16:12:51,643 INFO  assembly.AggregateBy (AggregateBy.java:removeEldestEntry(328)) - mem on flush (mb), free: 196, total: 511, max: 672
2015-02-12 16:12:51,643 INFO  assembly.AggregateBy (AggregateBy.java:removeEldestEntry(333)) - total mem is 76% of max mem, to better utilize unused memory consider increasing current LRU threshold with system property "cascading.aggregateby.threshold"
2015-02-12 16:12:51,643 INFO  assembly.AggregateBy (AggregateBy.java:removeEldestEntry(327)) - flushed keys num times: 7, with threshold: 2
2015-02-12 16:12:51,643 INFO  assembly.AggregateBy (AggregateBy.java:removeEldestEntry(328)) - mem on flush (mb), free: 196, total: 511, max: 672
2015-02-12 16:12:51,643 INFO  assembly.AggregateBy (AggregateBy.java:removeEldestEntry(333)) - total mem is 76% of max mem, to better utilize unused memory consider increasing current LRU threshold with system property "cascading.aggregateby.threshold"
2015-02-12 16:12:51,644 INFO  mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) - 
2015-02-12 16:12:51,644 INFO  mapred.MapTask (MapTask.java:flush(1457)) - Starting flush of map output
2015-02-12 16:12:51,644 INFO  mapred.MapTask (MapTask.java:flush(1475)) - Spilling map output
2015-02-12 16:12:51,644 INFO  mapred.MapTask (MapTask.java:flush(1476)) - bufstart = 0; bufend = 98; bufvoid = 104857600
2015-02-12 16:12:51,644 INFO  mapred.MapTask (MapTask.java:flush(1478)) - kvstart = 26214396(104857584); kvend = 26214364(104857456); length = 33/6553600
2015-02-12 16:12:51,646 INFO  mapred.MapTask (MapTask.java:sortAndSpill(1660)) - Finished spill 0
2015-02-12 16:12:51,647 INFO  mapred.Task (Task.java:done(1001)) - Task:attempt_local1388077210_0023_m_000000_0 is done. And is in the process of committing
2015-02-12 16:12:51,648 INFO  mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) - file:/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/classes/test/data/lhs.txt:0+52
2015-02-12 16:12:51,648 INFO  mapred.Task (Task.java:sendDone(1121)) - Task 'attempt_local1388077210_0023_m_000000_0' done.
2015-02-12 16:12:51,648 INFO  mapred.LocalJobRunner (LocalJobRunner.java:run(249)) - Finishing task: attempt_local1388077210_0023_m_000000_0
2015-02-12 16:12:51,648 INFO  mapred.LocalJobRunner (LocalJobRunner.java:runTasks(456)) - map task executor complete.
2015-02-12 16:12:51,649 INFO  mapred.LocalJobRunner (LocalJobRunner.java:runTasks(448)) - Waiting for reduce tasks
2015-02-12 16:12:51,649 INFO  mapred.LocalJobRunner (LocalJobRunner.java:run(302)) - Starting task: attempt_local1388077210_0023_r_000000_0
2015-02-12 16:12:51,650 INFO  util.ProcfsBasedProcessTree (ProcfsBasedProcessTree.java:isAvailable(181)) - ProcfsBasedProcessTree currently is supported only on Linux.
2015-02-12 16:12:51,650 INFO  mapred.Task (Task.java:initialize(587)) -  Using ResourceCalculatorProcessTree : null
2015-02-12 16:12:51,650 INFO  mapred.ReduceTask (ReduceTask.java:run(362)) - Using ShuffleConsumerPlugin: org.apache.hadoop.mapreduce.task.reduce.Shuffle@4f817cc9
2015-02-12 16:12:51,650 INFO  reduce.MergeManagerImpl (MergeManagerImpl.java:<init>(195)) - MergerManager: memoryLimit=493250144, maxSingleShuffleLimit=123312536, mergeThreshold=325545120, ioSortFactor=10, memToMemMergeOutputsThreshold=10
2015-02-12 16:12:51,651 INFO  reduce.EventFetcher (EventFetcher.java:run(61)) - attempt_local1388077210_0023_r_000000_0 Thread started: EventFetcher for fetching Map Completion Events
2015-02-12 16:12:51,652 INFO  reduce.LocalFetcher (LocalFetcher.java:copyMapOutput(141)) - localfetcher#19 about to shuffle output of map attempt_local1388077210_0023_m_000000_0 decomp: 118 len: 122 to MEMORY
2015-02-12 16:12:51,652 INFO  reduce.InMemoryMapOutput (InMemoryMapOutput.java:shuffle(100)) - Read 118 bytes from map-output for attempt_local1388077210_0023_m_000000_0
2015-02-12 16:12:51,652 INFO  reduce.MergeManagerImpl (MergeManagerImpl.java:closeInMemoryFile(313)) - closeInMemoryFile -> map-output of size: 118, inMemoryMapOutputs.size() -> 1, commitMemory -> 0, usedMemory ->118
2015-02-12 16:12:51,652 INFO  reduce.EventFetcher (EventFetcher.java:run(76)) - EventFetcher is interrupted.. Returning
2015-02-12 16:12:51,653 INFO  mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) - 1 / 1 copied.
2015-02-12 16:12:51,653 INFO  reduce.MergeManagerImpl (MergeManagerImpl.java:finalMerge(673)) - finalMerge called with 1 in-memory map-outputs and 0 on-disk map-outputs
2015-02-12 16:12:51,654 INFO  mapred.Merger (Merger.java:merge(606)) - Merging 1 sorted segments
2015-02-12 16:12:51,654 INFO  mapred.Merger (Merger.java:merge(705)) - Down to the last merge-pass, with 1 segments left of total size: 109 bytes
2015-02-12 16:12:51,655 INFO  reduce.MergeManagerImpl (MergeManagerImpl.java:finalMerge(750)) - Merged 1 segments, 118 bytes to disk to satisfy reduce memory limit
2015-02-12 16:12:51,655 INFO  reduce.MergeManagerImpl (MergeManagerImpl.java:finalMerge(780)) - Merging 1 files, 122 bytes from disk
2015-02-12 16:12:51,655 INFO  reduce.MergeManagerImpl (MergeManagerImpl.java:finalMerge(795)) - Merging 0 segments, 0 bytes from memory into reduce
2015-02-12 16:12:51,656 INFO  mapred.Merger (Merger.java:merge(606)) - Merging 1 sorted segments
2015-02-12 16:12:51,656 INFO  mapred.Merger (Merger.java:merge(705)) - Down to the last merge-pass, with 1 segments left of total size: 109 bytes
2015-02-12 16:12:51,656 INFO  mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) - 1 / 1 copied.
2015-02-12 16:12:51,657 INFO  hadoop.FlowReducer (FlowReducer.java:configure(78)) - cascading version: 2.6.3
2015-02-12 16:12:51,657 INFO  hadoop.FlowReducer (FlowReducer.java:configure(79)) - child jvm opts: -Xmx200m
2015-02-12 16:12:51,661 INFO  hadoop.FlowReducer (FlowReducer.java:configure(97)) - sourcing from: GroupBy(max)[by:[{1}:'char']]
2015-02-12 16:12:51,661 INFO  hadoop.FlowReducer (FlowReducer.java:configure(100)) - sinking to: Hfs["TextDelimited[['char', 'max']]"]["/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/test/output/hadoop2-mr1/assemblyhelpersplatform/maxbynullsafety"]
2015-02-12 16:12:51,668 INFO  mapred.Task (Task.java:done(1001)) - Task:attempt_local1388077210_0023_r_000000_0 is done. And is in the process of committing
2015-02-12 16:12:51,668 INFO  mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) - 1 / 1 copied.
2015-02-12 16:12:51,669 INFO  mapred.Task (Task.java:commit(1162)) - Task attempt_local1388077210_0023_r_000000_0 is allowed to commit now
2015-02-12 16:12:51,669 INFO  output.FileOutputCommitter (FileOutputCommitter.java:commitTask(439)) - Saved output of task 'attempt_local1388077210_0023_r_000000_0' to file:/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/test/output/hadoop2-mr1/assemblyhelpersplatform/maxbynullsafety/_temporary/0/task_local1388077210_0023_r_000000
2015-02-12 16:12:51,670 INFO  mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) - reduce > reduce
2015-02-12 16:12:51,670 INFO  mapred.Task (Task.java:sendDone(1121)) - Task 'attempt_local1388077210_0023_r_000000_0' done.
2015-02-12 16:12:51,670 INFO  mapred.LocalJobRunner (LocalJobRunner.java:run(325)) - Finishing task: attempt_local1388077210_0023_r_000000_0
2015-02-12 16:12:51,670 INFO  mapred.LocalJobRunner (LocalJobRunner.java:runTasks(456)) - reduce task executor complete.
2015-02-12 16:12:51,688 INFO  util.Hadoop18TapUtil (Hadoop18TapUtil.java:cleanTempPath(219)) - deleting temp path /Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/test/output/hadoop2-mr1/assemblyhelpersplatform/maxbynullsafety/_temporary
2015-02-12 16:12:51,708 INFO  mapred.FileInputFormat (FileInputFormat.java:listStatus(247)) - Total input paths to process : 1
2015-02-12 16:12:51,732 INFO  mapred.FileInputFormat (FileInputFormat.java:listStatus(247)) - Total input paths to process : 1
2015-02-12 16:12:51,733 INFO  cascading.PlatformTestCase (PlatformTestCase.java:tearDown(209)) - copying to local /Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/test/output//hadoop2-mr1/assemblyhelpersplatform/maxbynullsafety
2015-02-12 16:12:51,744 INFO  util.HadoopUtil (HadoopUtil.java:findMainClass(336)) - using default application jar, may cause class not found exceptions on the cluster
2015-02-12 16:12:51,744 INFO  planner.HadoopPlanner (HadoopPlanner.java:initialize(225)) - using application jar: /Users/soberoi/.gradle/caches/modules-2/files-2.1/cascading/cascading-hadoop2-mr1/2.6.3/9ae016e2b8d246b9c1de9168e715d9f1b971aa4a/cascading-hadoop2-mr1-2.6.3.jar
2015-02-12 16:12:51,748 INFO  Configuration.deprecation (Configuration.java:warnOnceIfDeprecated(1022)) - mapred.used.genericoptionsparser is deprecated. Instead, use mapreduce.client.genericoptionsparser.used
2015-02-12 16:12:51,749 INFO  Configuration.deprecation (Configuration.java:warnOnceIfDeprecated(1022)) - mapred.job.tracker is deprecated. Instead, use mapreduce.jobtracker.address
2015-02-12 16:12:51,749 INFO  Configuration.deprecation (Configuration.java:warnOnceIfDeprecated(1022)) - mapred.output.compress is deprecated. Instead, use mapreduce.output.fileoutputformat.compress
2015-02-12 16:12:51,754 INFO  flow.Flow (BaseFlow.java:logInfo(1378)) - [sum] starting
2015-02-12 16:12:51,754 INFO  flow.Flow (BaseFlow.java:logInfo(1378)) - [sum]  source: Hfs["TextDelimited[['num', 'char']]"]["/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/classes/test/data/lhs.txt"]
2015-02-12 16:12:51,754 INFO  flow.Flow (BaseFlow.java:logInfo(1378)) - [sum]  sink: Hfs["TextDelimited[['char', 'sum']]"]["/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/test/output/hadoop2-mr1/assemblyhelpersplatform/sumnulls"]
2015-02-12 16:12:51,754 INFO  flow.Flow (BaseFlow.java:logInfo(1378)) - [sum]  parallel execution is enabled: false
2015-02-12 16:12:51,754 INFO  flow.Flow (BaseFlow.java:logInfo(1378)) - [sum]  starting jobs: 1
2015-02-12 16:12:51,754 INFO  flow.Flow (BaseFlow.java:logInfo(1378)) - [sum]  allocating threads: 1
2015-02-12 16:12:51,755 INFO  flow.FlowStep (BaseFlowStep.java:logInfo(834)) - [sum] starting step: (1/1) ...yhelpersplatform/sumnulls
2015-02-12 16:12:51,756 INFO  jvm.JvmMetrics (JvmMetrics.java:init(71)) - Cannot initialize JVM Metrics with processName=JobTracker, sessionId= - already initialized
2015-02-12 16:12:51,756 INFO  jvm.JvmMetrics (JvmMetrics.java:init(71)) - Cannot initialize JVM Metrics with processName=JobTracker, sessionId= - already initialized
2015-02-12 16:12:51,775 INFO  mapred.FileInputFormat (FileInputFormat.java:listStatus(247)) - Total input paths to process : 1
2015-02-12 16:12:51,791 INFO  mapreduce.JobSubmitter (JobSubmitter.java:submitJobInternal(492)) - number of splits:1
2015-02-12 16:12:51,802 INFO  mapreduce.JobSubmitter (JobSubmitter.java:printTokens(575)) - Submitting tokens for job: job_local1795518538_0024
2015-02-12 16:12:51,813 WARN  conf.Configuration (Configuration.java:loadProperty(2510)) - file:/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/tmp/cascading/staging/soberoi1795518538/.staging/job_local1795518538_0024/job.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.retry.interval;  Ignoring.
2015-02-12 16:12:51,814 WARN  conf.Configuration (Configuration.java:loadProperty(2510)) - file:/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/tmp/cascading/staging/soberoi1795518538/.staging/job_local1795518538_0024/job.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.attempts;  Ignoring.
2015-02-12 16:12:51,846 WARN  conf.Configuration (Configuration.java:loadProperty(2510)) - file:/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/test/tmp/mapred/local/localRunner/soberoi/job_local1795518538_0024/job_local1795518538_0024.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.retry.interval;  Ignoring.
2015-02-12 16:12:51,847 WARN  conf.Configuration (Configuration.java:loadProperty(2510)) - file:/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/test/tmp/mapred/local/localRunner/soberoi/job_local1795518538_0024/job_local1795518538_0024.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.attempts;  Ignoring.
2015-02-12 16:12:51,847 INFO  mapreduce.Job (Job.java:submit(1299)) - The url to track the job: http://localhost:8080/
2015-02-12 16:12:51,847 INFO  mapred.LocalJobRunner (LocalJobRunner.java:createOutputCommitter(471)) - OutputCommitter set in config null
2015-02-12 16:12:51,847 INFO  flow.FlowStep (BaseFlowStep.java:logInfo(834)) - [sum] submitted hadoop job: job_local1795518538_0024
2015-02-12 16:12:51,847 INFO  mapred.LocalJobRunner (LocalJobRunner.java:createOutputCommitter(489)) - OutputCommitter is org.apache.hadoop.mapred.FileOutputCommitter
2015-02-12 16:12:51,847 INFO  flow.FlowStep (BaseFlowStep.java:logInfo(834)) - [sum] tracking url: http://localhost:8080/
2015-02-12 16:12:51,849 INFO  mapred.LocalJobRunner (LocalJobRunner.java:runTasks(448)) - Waiting for map tasks
2015-02-12 16:12:51,849 INFO  mapred.LocalJobRunner (LocalJobRunner.java:run(224)) - Starting task: attempt_local1795518538_0024_m_000000_0
2015-02-12 16:12:51,850 INFO  util.ProcfsBasedProcessTree (ProcfsBasedProcessTree.java:isAvailable(181)) - ProcfsBasedProcessTree currently is supported only on Linux.
2015-02-12 16:12:51,850 INFO  mapred.Task (Task.java:initialize(587)) -  Using ResourceCalculatorProcessTree : null
2015-02-12 16:12:51,851 INFO  io.MultiInputSplit (MultiInputSplit.java:readFields(161)) - current split input path: file:/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/classes/test/data/lhs.txt
2015-02-12 16:12:51,851 INFO  mapred.MapTask (MapTask.java:updateJobWithSplit(482)) - Processing split: cascading.tap.hadoop.io.MultiInputSplit@378577ad
2015-02-12 16:12:51,851 INFO  mapred.MapTask (MapTask.java:runOldMapper(436)) - numReduceTasks: 1
2015-02-12 16:12:51,867 INFO  mapred.MapTask (MapTask.java:setEquator(1202)) - (EQUATOR) 0 kvi 26214396(104857584)
2015-02-12 16:12:51,867 INFO  mapred.MapTask (MapTask.java:init(995)) - mapreduce.task.io.sort.mb: 100
2015-02-12 16:12:51,867 INFO  mapred.MapTask (MapTask.java:init(996)) - soft limit at 83886080
2015-02-12 16:12:51,867 INFO  mapred.MapTask (MapTask.java:init(997)) - bufstart = 0; bufvoid = 104857600
2015-02-12 16:12:51,867 INFO  mapred.MapTask (MapTask.java:init(998)) - kvstart = 26214396; length = 6553600
2015-02-12 16:12:51,868 INFO  mapred.MapTask (MapTask.java:createSortingCollector(402)) - Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer
2015-02-12 16:12:51,868 INFO  hadoop.FlowMapper (FlowMapper.java:configure(71)) - cascading version: 2.6.3
2015-02-12 16:12:51,868 INFO  hadoop.FlowMapper (FlowMapper.java:configure(72)) - child jvm opts: -Xmx200m
2015-02-12 16:12:51,871 INFO  hadoop.FlowMapper (FlowMapper.java:configure(87)) - sourcing from: Hfs["TextDelimited[['num', 'char']]"]["/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/classes/test/data/lhs.txt"]
2015-02-12 16:12:51,871 INFO  hadoop.FlowMapper (FlowMapper.java:configure(90)) - sinking to: GroupBy(sum)[by:[{1}:'char']]
2015-02-12 16:12:51,872 INFO  assembly.AggregateBy (AggregateBy.java:prepare(267)) - using threshold value: 2
2015-02-12 16:12:51,876 INFO  assembly.AggregateBy (AggregateBy.java:removeEldestEntry(327)) - flushed keys num times: 1, with threshold: 2
2015-02-12 16:12:51,876 INFO  assembly.AggregateBy (AggregateBy.java:removeEldestEntry(328)) - mem on flush (mb), free: 196, total: 512, max: 672
2015-02-12 16:12:51,876 INFO  assembly.AggregateBy (AggregateBy.java:removeEldestEntry(333)) - total mem is 76% of max mem, to better utilize unused memory consider increasing current LRU threshold with system property "cascading.aggregateby.threshold"
2015-02-12 16:12:51,877 INFO  assembly.AggregateBy (AggregateBy.java:removeEldestEntry(327)) - flushed keys num times: 3, with threshold: 2
2015-02-12 16:12:51,877 INFO  assembly.AggregateBy (AggregateBy.java:removeEldestEntry(328)) - mem on flush (mb), free: 196, total: 512, max: 672
2015-02-12 16:12:51,877 INFO  assembly.AggregateBy (AggregateBy.java:removeEldestEntry(333)) - total mem is 76% of max mem, to better utilize unused memory consider increasing current LRU threshold with system property "cascading.aggregateby.threshold"
2015-02-12 16:12:51,878 INFO  assembly.AggregateBy (AggregateBy.java:removeEldestEntry(327)) - flushed keys num times: 5, with threshold: 2
2015-02-12 16:12:51,878 INFO  assembly.AggregateBy (AggregateBy.java:removeEldestEntry(328)) - mem on flush (mb), free: 196, total: 512, max: 672
2015-02-12 16:12:51,878 INFO  assembly.AggregateBy (AggregateBy.java:removeEldestEntry(333)) - total mem is 76% of max mem, to better utilize unused memory consider increasing current LRU threshold with system property "cascading.aggregateby.threshold"
2015-02-12 16:12:51,878 INFO  assembly.AggregateBy (AggregateBy.java:removeEldestEntry(327)) - flushed keys num times: 7, with threshold: 2
2015-02-12 16:12:51,878 INFO  assembly.AggregateBy (AggregateBy.java:removeEldestEntry(328)) - mem on flush (mb), free: 196, total: 512, max: 672
2015-02-12 16:12:51,878 INFO  assembly.AggregateBy (AggregateBy.java:removeEldestEntry(333)) - total mem is 76% of max mem, to better utilize unused memory consider increasing current LRU threshold with system property "cascading.aggregateby.threshold"
2015-02-12 16:12:51,879 INFO  mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) - 
2015-02-12 16:12:51,879 INFO  mapred.MapTask (MapTask.java:flush(1457)) - Starting flush of map output
2015-02-12 16:12:51,879 INFO  mapred.MapTask (MapTask.java:flush(1475)) - Spilling map output
2015-02-12 16:12:51,880 INFO  mapred.MapTask (MapTask.java:flush(1476)) - bufstart = 0; bufend = 96; bufvoid = 104857600
2015-02-12 16:12:51,880 INFO  mapred.MapTask (MapTask.java:flush(1478)) - kvstart = 26214396(104857584); kvend = 26214364(104857456); length = 33/6553600
2015-02-12 16:12:51,881 INFO  mapred.MapTask (MapTask.java:sortAndSpill(1660)) - Finished spill 0
2015-02-12 16:12:51,883 INFO  mapred.Task (Task.java:done(1001)) - Task:attempt_local1795518538_0024_m_000000_0 is done. And is in the process of committing
2015-02-12 16:12:51,884 INFO  mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) - file:/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/classes/test/data/lhs.txt:0+52
2015-02-12 16:12:51,884 INFO  mapred.Task (Task.java:sendDone(1121)) - Task 'attempt_local1795518538_0024_m_000000_0' done.
2015-02-12 16:12:51,884 INFO  mapred.LocalJobRunner (LocalJobRunner.java:run(249)) - Finishing task: attempt_local1795518538_0024_m_000000_0
2015-02-12 16:12:51,884 INFO  mapred.LocalJobRunner (LocalJobRunner.java:runTasks(456)) - map task executor complete.
2015-02-12 16:12:51,884 INFO  mapred.LocalJobRunner (LocalJobRunner.java:runTasks(448)) - Waiting for reduce tasks
2015-02-12 16:12:51,884 INFO  mapred.LocalJobRunner (LocalJobRunner.java:run(302)) - Starting task: attempt_local1795518538_0024_r_000000_0
2015-02-12 16:12:51,885 INFO  util.ProcfsBasedProcessTree (ProcfsBasedProcessTree.java:isAvailable(181)) - ProcfsBasedProcessTree currently is supported only on Linux.
2015-02-12 16:12:51,886 INFO  mapred.Task (Task.java:initialize(587)) -  Using ResourceCalculatorProcessTree : null
2015-02-12 16:12:51,886 INFO  mapred.ReduceTask (ReduceTask.java:run(362)) - Using ShuffleConsumerPlugin: org.apache.hadoop.mapreduce.task.reduce.Shuffle@7ac7e297
2015-02-12 16:12:51,886 INFO  reduce.MergeManagerImpl (MergeManagerImpl.java:<init>(195)) - MergerManager: memoryLimit=493250144, maxSingleShuffleLimit=123312536, mergeThreshold=325545120, ioSortFactor=10, memToMemMergeOutputsThreshold=10
2015-02-12 16:12:51,886 INFO  reduce.EventFetcher (EventFetcher.java:run(61)) - attempt_local1795518538_0024_r_000000_0 Thread started: EventFetcher for fetching Map Completion Events
2015-02-12 16:12:51,887 INFO  reduce.LocalFetcher (LocalFetcher.java:copyMapOutput(141)) - localfetcher#20 about to shuffle output of map attempt_local1795518538_0024_m_000000_0 decomp: 116 len: 120 to MEMORY
2015-02-12 16:12:51,888 INFO  reduce.InMemoryMapOutput (InMemoryMapOutput.java:shuffle(100)) - Read 116 bytes from map-output for attempt_local1795518538_0024_m_000000_0
2015-02-12 16:12:51,888 INFO  reduce.MergeManagerImpl (MergeManagerImpl.java:closeInMemoryFile(313)) - closeInMemoryFile -> map-output of size: 116, inMemoryMapOutputs.size() -> 1, commitMemory -> 0, usedMemory ->116
2015-02-12 16:12:51,888 INFO  reduce.EventFetcher (EventFetcher.java:run(76)) - EventFetcher is interrupted.. Returning
2015-02-12 16:12:51,889 INFO  mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) - 1 / 1 copied.
2015-02-12 16:12:51,889 INFO  reduce.MergeManagerImpl (MergeManagerImpl.java:finalMerge(673)) - finalMerge called with 1 in-memory map-outputs and 0 on-disk map-outputs
2015-02-12 16:12:51,890 INFO  mapred.Merger (Merger.java:merge(606)) - Merging 1 sorted segments
2015-02-12 16:12:51,890 INFO  mapred.Merger (Merger.java:merge(705)) - Down to the last merge-pass, with 1 segments left of total size: 107 bytes
2015-02-12 16:12:51,891 INFO  reduce.MergeManagerImpl (MergeManagerImpl.java:finalMerge(750)) - Merged 1 segments, 116 bytes to disk to satisfy reduce memory limit
2015-02-12 16:12:51,891 INFO  reduce.MergeManagerImpl (MergeManagerImpl.java:finalMerge(780)) - Merging 1 files, 120 bytes from disk
2015-02-12 16:12:51,891 INFO  reduce.MergeManagerImpl (MergeManagerImpl.java:finalMerge(795)) - Merging 0 segments, 0 bytes from memory into reduce
2015-02-12 16:12:51,892 INFO  mapred.Merger (Merger.java:merge(606)) - Merging 1 sorted segments
2015-02-12 16:12:51,892 INFO  mapred.Merger (Merger.java:merge(705)) - Down to the last merge-pass, with 1 segments left of total size: 107 bytes
2015-02-12 16:12:51,892 INFO  mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) - 1 / 1 copied.
2015-02-12 16:12:51,893 INFO  hadoop.FlowReducer (FlowReducer.java:configure(78)) - cascading version: 2.6.3
2015-02-12 16:12:51,893 INFO  hadoop.FlowReducer (FlowReducer.java:configure(79)) - child jvm opts: -Xmx200m
2015-02-12 16:12:51,897 INFO  hadoop.FlowReducer (FlowReducer.java:configure(97)) - sourcing from: GroupBy(sum)[by:[{1}:'char']]
2015-02-12 16:12:51,897 INFO  hadoop.FlowReducer (FlowReducer.java:configure(100)) - sinking to: Hfs["TextDelimited[['char', 'sum']]"]["/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/test/output/hadoop2-mr1/assemblyhelpersplatform/sumnulls"]
2015-02-12 16:12:51,903 INFO  mapred.Task (Task.java:done(1001)) - Task:attempt_local1795518538_0024_r_000000_0 is done. And is in the process of committing
2015-02-12 16:12:51,904 INFO  mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) - 1 / 1 copied.
2015-02-12 16:12:51,904 INFO  mapred.Task (Task.java:commit(1162)) - Task attempt_local1795518538_0024_r_000000_0 is allowed to commit now
2015-02-12 16:12:51,905 INFO  output.FileOutputCommitter (FileOutputCommitter.java:commitTask(439)) - Saved output of task 'attempt_local1795518538_0024_r_000000_0' to file:/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/test/output/hadoop2-mr1/assemblyhelpersplatform/sumnulls/_temporary/0/task_local1795518538_0024_r_000000
2015-02-12 16:12:51,905 INFO  mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) - reduce > reduce
2015-02-12 16:12:51,905 INFO  mapred.Task (Task.java:sendDone(1121)) - Task 'attempt_local1795518538_0024_r_000000_0' done.
2015-02-12 16:12:51,905 INFO  mapred.LocalJobRunner (LocalJobRunner.java:run(325)) - Finishing task: attempt_local1795518538_0024_r_000000_0
2015-02-12 16:12:51,905 INFO  mapred.LocalJobRunner (LocalJobRunner.java:runTasks(456)) - reduce task executor complete.
2015-02-12 16:12:51,927 INFO  util.Hadoop18TapUtil (Hadoop18TapUtil.java:cleanTempPath(219)) - deleting temp path /Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/test/output/hadoop2-mr1/assemblyhelpersplatform/sumnulls/_temporary
2015-02-12 16:12:51,946 INFO  mapred.FileInputFormat (FileInputFormat.java:listStatus(247)) - Total input paths to process : 1
2015-02-12 16:12:51,966 INFO  mapred.FileInputFormat (FileInputFormat.java:listStatus(247)) - Total input paths to process : 1
2015-02-12 16:12:51,967 INFO  cascading.PlatformTestCase (PlatformTestCase.java:tearDown(209)) - copying to local /Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/test/output//hadoop2-mr1/assemblyhelpersplatform/sumnulls
2015-02-12 16:12:51,981 INFO  util.HadoopUtil (HadoopUtil.java:findMainClass(336)) - using default application jar, may cause class not found exceptions on the cluster
2015-02-12 16:12:51,981 INFO  planner.HadoopPlanner (HadoopPlanner.java:initialize(225)) - using application jar: /Users/soberoi/.gradle/caches/modules-2/files-2.1/cascading/cascading-hadoop2-mr1/2.6.3/9ae016e2b8d246b9c1de9168e715d9f1b971aa4a/cascading-hadoop2-mr1-2.6.3.jar
2015-02-12 16:12:51,984 INFO  Configuration.deprecation (Configuration.java:warnOnceIfDeprecated(1022)) - mapred.used.genericoptionsparser is deprecated. Instead, use mapreduce.client.genericoptionsparser.used
2015-02-12 16:12:51,985 INFO  Configuration.deprecation (Configuration.java:warnOnceIfDeprecated(1022)) - mapred.job.tracker is deprecated. Instead, use mapreduce.jobtracker.address
2015-02-12 16:12:51,985 INFO  Configuration.deprecation (Configuration.java:warnOnceIfDeprecated(1022)) - mapred.output.compress is deprecated. Instead, use mapreduce.output.fileoutputformat.compress
2015-02-12 16:12:51,989 INFO  flow.Flow (BaseFlow.java:logInfo(1378)) - [count] starting
2015-02-12 16:12:51,990 INFO  flow.Flow (BaseFlow.java:logInfo(1378)) - [count]  source: Hfs["TextDelimited[['num', 'char']]"]["/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/classes/test/data/lhs.txt"]
2015-02-12 16:12:51,990 INFO  flow.Flow (BaseFlow.java:logInfo(1378)) - [count]  sink: Hfs["TextDelimited[['count']]"]["/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/test/output/hadoop2-mr1/assemblyhelpersplatform/countall"]
2015-02-12 16:12:51,990 INFO  flow.Flow (BaseFlow.java:logInfo(1378)) - [count]  parallel execution is enabled: false
2015-02-12 16:12:51,990 INFO  flow.Flow (BaseFlow.java:logInfo(1378)) - [count]  starting jobs: 1
2015-02-12 16:12:51,990 INFO  flow.Flow (BaseFlow.java:logInfo(1378)) - [count]  allocating threads: 1
2015-02-12 16:12:51,990 INFO  flow.FlowStep (BaseFlowStep.java:logInfo(834)) - [count] starting step: (1/1) ...yhelpersplatform/countall
2015-02-12 16:12:51,991 INFO  jvm.JvmMetrics (JvmMetrics.java:init(71)) - Cannot initialize JVM Metrics with processName=JobTracker, sessionId= - already initialized
2015-02-12 16:12:51,991 INFO  jvm.JvmMetrics (JvmMetrics.java:init(71)) - Cannot initialize JVM Metrics with processName=JobTracker, sessionId= - already initialized
2015-02-12 16:12:52,010 INFO  mapred.FileInputFormat (FileInputFormat.java:listStatus(247)) - Total input paths to process : 1
2015-02-12 16:12:52,025 INFO  mapreduce.JobSubmitter (JobSubmitter.java:submitJobInternal(492)) - number of splits:1
2015-02-12 16:12:52,037 INFO  mapreduce.JobSubmitter (JobSubmitter.java:printTokens(575)) - Submitting tokens for job: job_local375512920_0025
2015-02-12 16:12:52,048 WARN  conf.Configuration (Configuration.java:loadProperty(2510)) - file:/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/tmp/cascading/staging/soberoi375512920/.staging/job_local375512920_0025/job.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.retry.interval;  Ignoring.
2015-02-12 16:12:52,049 WARN  conf.Configuration (Configuration.java:loadProperty(2510)) - file:/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/tmp/cascading/staging/soberoi375512920/.staging/job_local375512920_0025/job.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.attempts;  Ignoring.
2015-02-12 16:12:52,078 WARN  conf.Configuration (Configuration.java:loadProperty(2510)) - file:/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/test/tmp/mapred/local/localRunner/soberoi/job_local375512920_0025/job_local375512920_0025.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.retry.interval;  Ignoring.
2015-02-12 16:12:52,079 WARN  conf.Configuration (Configuration.java:loadProperty(2510)) - file:/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/test/tmp/mapred/local/localRunner/soberoi/job_local375512920_0025/job_local375512920_0025.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.attempts;  Ignoring.
2015-02-12 16:12:52,080 INFO  mapreduce.Job (Job.java:submit(1299)) - The url to track the job: http://localhost:8080/
2015-02-12 16:12:52,080 INFO  flow.FlowStep (BaseFlowStep.java:logInfo(834)) - [count] submitted hadoop job: job_local375512920_0025
2015-02-12 16:12:52,080 INFO  mapred.LocalJobRunner (LocalJobRunner.java:createOutputCommitter(471)) - OutputCommitter set in config null
2015-02-12 16:12:52,080 INFO  flow.FlowStep (BaseFlowStep.java:logInfo(834)) - [count] tracking url: http://localhost:8080/
2015-02-12 16:12:52,081 INFO  mapred.LocalJobRunner (LocalJobRunner.java:createOutputCommitter(489)) - OutputCommitter is org.apache.hadoop.mapred.FileOutputCommitter
2015-02-12 16:12:52,082 INFO  mapred.LocalJobRunner (LocalJobRunner.java:runTasks(448)) - Waiting for map tasks
2015-02-12 16:12:52,082 INFO  mapred.LocalJobRunner (LocalJobRunner.java:run(224)) - Starting task: attempt_local375512920_0025_m_000000_0
2015-02-12 16:12:52,083 INFO  util.ProcfsBasedProcessTree (ProcfsBasedProcessTree.java:isAvailable(181)) - ProcfsBasedProcessTree currently is supported only on Linux.
2015-02-12 16:12:52,083 INFO  mapred.Task (Task.java:initialize(587)) -  Using ResourceCalculatorProcessTree : null
2015-02-12 16:12:52,084 INFO  io.MultiInputSplit (MultiInputSplit.java:readFields(161)) - current split input path: file:/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/classes/test/data/lhs.txt
2015-02-12 16:12:52,084 INFO  mapred.MapTask (MapTask.java:updateJobWithSplit(482)) - Processing split: cascading.tap.hadoop.io.MultiInputSplit@7433ccf0
2015-02-12 16:12:52,085 INFO  mapred.MapTask (MapTask.java:runOldMapper(436)) - numReduceTasks: 1
2015-02-12 16:12:52,101 INFO  mapred.MapTask (MapTask.java:setEquator(1202)) - (EQUATOR) 0 kvi 26214396(104857584)
2015-02-12 16:12:52,101 INFO  mapred.MapTask (MapTask.java:init(995)) - mapreduce.task.io.sort.mb: 100
2015-02-12 16:12:52,101 INFO  mapred.MapTask (MapTask.java:init(996)) - soft limit at 83886080
2015-02-12 16:12:52,101 INFO  mapred.MapTask (MapTask.java:init(997)) - bufstart = 0; bufvoid = 104857600
2015-02-12 16:12:52,101 INFO  mapred.MapTask (MapTask.java:init(998)) - kvstart = 26214396; length = 6553600
2015-02-12 16:12:52,102 INFO  mapred.MapTask (MapTask.java:createSortingCollector(402)) - Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer
2015-02-12 16:12:52,102 INFO  hadoop.FlowMapper (FlowMapper.java:configure(71)) - cascading version: 2.6.3
2015-02-12 16:12:52,102 INFO  hadoop.FlowMapper (FlowMapper.java:configure(72)) - child jvm opts: -Xmx200m
2015-02-12 16:12:52,106 INFO  hadoop.FlowMapper (FlowMapper.java:configure(87)) - sourcing from: Hfs["TextDelimited[['num', 'char']]"]["/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/classes/test/data/lhs.txt"]
2015-02-12 16:12:52,106 INFO  hadoop.FlowMapper (FlowMapper.java:configure(90)) - sinking to: GroupBy(count)[by:[{?}:NONE]]
2015-02-12 16:12:52,107 INFO  assembly.AggregateBy (AggregateBy.java:prepare(267)) - using threshold value: 2
2015-02-12 16:12:52,109 INFO  mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) - 
2015-02-12 16:12:52,109 INFO  mapred.MapTask (MapTask.java:flush(1457)) - Starting flush of map output
2015-02-12 16:12:52,109 INFO  mapred.MapTask (MapTask.java:flush(1475)) - Spilling map output
2015-02-12 16:12:52,109 INFO  mapred.MapTask (MapTask.java:flush(1476)) - bufstart = 0; bufend = 4; bufvoid = 104857600
2015-02-12 16:12:52,109 INFO  mapred.MapTask (MapTask.java:flush(1478)) - kvstart = 26214396(104857584); kvend = 26214396(104857584); length = 1/6553600
2015-02-12 16:12:52,112 INFO  mapred.MapTask (MapTask.java:sortAndSpill(1660)) - Finished spill 0
2015-02-12 16:12:52,114 INFO  mapred.Task (Task.java:done(1001)) - Task:attempt_local375512920_0025_m_000000_0 is done. And is in the process of committing
2015-02-12 16:12:52,115 INFO  mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) - file:/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/classes/test/data/lhs.txt:0+52
2015-02-12 16:12:52,116 INFO  mapred.Task (Task.java:sendDone(1121)) - Task 'attempt_local375512920_0025_m_000000_0' done.
2015-02-12 16:12:52,116 INFO  mapred.LocalJobRunner (LocalJobRunner.java:run(249)) - Finishing task: attempt_local375512920_0025_m_000000_0
2015-02-12 16:12:52,116 INFO  mapred.LocalJobRunner (LocalJobRunner.java:runTasks(456)) - map task executor complete.
2015-02-12 16:12:52,116 INFO  mapred.LocalJobRunner (LocalJobRunner.java:runTasks(448)) - Waiting for reduce tasks
2015-02-12 16:12:52,116 INFO  mapred.LocalJobRunner (LocalJobRunner.java:run(302)) - Starting task: attempt_local375512920_0025_r_000000_0
2015-02-12 16:12:52,118 INFO  util.ProcfsBasedProcessTree (ProcfsBasedProcessTree.java:isAvailable(181)) - ProcfsBasedProcessTree currently is supported only on Linux.
2015-02-12 16:12:52,118 INFO  mapred.Task (Task.java:initialize(587)) -  Using ResourceCalculatorProcessTree : null
2015-02-12 16:12:52,118 INFO  mapred.ReduceTask (ReduceTask.java:run(362)) - Using ShuffleConsumerPlugin: org.apache.hadoop.mapreduce.task.reduce.Shuffle@42ae638
2015-02-12 16:12:52,118 INFO  reduce.MergeManagerImpl (MergeManagerImpl.java:<init>(195)) - MergerManager: memoryLimit=493250144, maxSingleShuffleLimit=123312536, mergeThreshold=325545120, ioSortFactor=10, memToMemMergeOutputsThreshold=10
2015-02-12 16:12:52,119 INFO  reduce.EventFetcher (EventFetcher.java:run(61)) - attempt_local375512920_0025_r_000000_0 Thread started: EventFetcher for fetching Map Completion Events
2015-02-12 16:12:52,120 INFO  reduce.LocalFetcher (LocalFetcher.java:copyMapOutput(141)) - localfetcher#21 about to shuffle output of map attempt_local375512920_0025_m_000000_0 decomp: 8 len: 12 to MEMORY
2015-02-12 16:12:52,120 INFO  reduce.InMemoryMapOutput (InMemoryMapOutput.java:shuffle(100)) - Read 8 bytes from map-output for attempt_local375512920_0025_m_000000_0
2015-02-12 16:12:52,120 INFO  reduce.MergeManagerImpl (MergeManagerImpl.java:closeInMemoryFile(313)) - closeInMemoryFile -> map-output of size: 8, inMemoryMapOutputs.size() -> 1, commitMemory -> 0, usedMemory ->8
2015-02-12 16:12:52,121 INFO  reduce.EventFetcher (EventFetcher.java:run(76)) - EventFetcher is interrupted.. Returning
2015-02-12 16:12:52,121 INFO  mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) - 1 / 1 copied.
2015-02-12 16:12:52,122 INFO  reduce.MergeManagerImpl (MergeManagerImpl.java:finalMerge(673)) - finalMerge called with 1 in-memory map-outputs and 0 on-disk map-outputs
2015-02-12 16:12:52,123 INFO  mapred.Merger (Merger.java:merge(606)) - Merging 1 sorted segments
2015-02-12 16:12:52,123 INFO  mapred.Merger (Merger.java:merge(705)) - Down to the last merge-pass, with 1 segments left of total size: 5 bytes
2015-02-12 16:12:52,124 INFO  reduce.MergeManagerImpl (MergeManagerImpl.java:finalMerge(750)) - Merged 1 segments, 8 bytes to disk to satisfy reduce memory limit
2015-02-12 16:12:52,125 INFO  reduce.MergeManagerImpl (MergeManagerImpl.java:finalMerge(780)) - Merging 1 files, 12 bytes from disk
2015-02-12 16:12:52,125 INFO  reduce.MergeManagerImpl (MergeManagerImpl.java:finalMerge(795)) - Merging 0 segments, 0 bytes from memory into reduce
2015-02-12 16:12:52,125 INFO  mapred.Merger (Merger.java:merge(606)) - Merging 1 sorted segments
2015-02-12 16:12:52,125 INFO  mapred.Merger (Merger.java:merge(705)) - Down to the last merge-pass, with 1 segments left of total size: 5 bytes
2015-02-12 16:12:52,126 INFO  mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) - 1 / 1 copied.
2015-02-12 16:12:52,126 INFO  hadoop.FlowReducer (FlowReducer.java:configure(78)) - cascading version: 2.6.3
2015-02-12 16:12:52,126 INFO  hadoop.FlowReducer (FlowReducer.java:configure(79)) - child jvm opts: -Xmx200m
2015-02-12 16:12:52,132 INFO  hadoop.FlowReducer (FlowReducer.java:configure(97)) - sourcing from: GroupBy(count)[by:[{?}:NONE]]
2015-02-12 16:12:52,132 INFO  hadoop.FlowReducer (FlowReducer.java:configure(100)) - sinking to: Hfs["TextDelimited[['count']]"]["/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/test/output/hadoop2-mr1/assemblyhelpersplatform/countall"]
2015-02-12 16:12:52,140 INFO  mapred.Task (Task.java:done(1001)) - Task:attempt_local375512920_0025_r_000000_0 is done. And is in the process of committing
2015-02-12 16:12:52,141 INFO  mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) - 1 / 1 copied.
2015-02-12 16:12:52,142 INFO  mapred.Task (Task.java:commit(1162)) - Task attempt_local375512920_0025_r_000000_0 is allowed to commit now
2015-02-12 16:12:52,142 INFO  output.FileOutputCommitter (FileOutputCommitter.java:commitTask(439)) - Saved output of task 'attempt_local375512920_0025_r_000000_0' to file:/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/test/output/hadoop2-mr1/assemblyhelpersplatform/countall/_temporary/0/task_local375512920_0025_r_000000
2015-02-12 16:12:52,143 INFO  mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) - reduce > reduce
2015-02-12 16:12:52,143 INFO  mapred.Task (Task.java:sendDone(1121)) - Task 'attempt_local375512920_0025_r_000000_0' done.
2015-02-12 16:12:52,143 INFO  mapred.LocalJobRunner (LocalJobRunner.java:run(325)) - Finishing task: attempt_local375512920_0025_r_000000_0
2015-02-12 16:12:52,143 INFO  mapred.LocalJobRunner (LocalJobRunner.java:runTasks(456)) - reduce task executor complete.
2015-02-12 16:12:52,164 INFO  util.Hadoop18TapUtil (Hadoop18TapUtil.java:cleanTempPath(219)) - deleting temp path /Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/test/output/hadoop2-mr1/assemblyhelpersplatform/countall/_temporary
2015-02-12 16:12:52,184 INFO  mapred.FileInputFormat (FileInputFormat.java:listStatus(247)) - Total input paths to process : 1
2015-02-12 16:12:52,205 INFO  mapred.FileInputFormat (FileInputFormat.java:listStatus(247)) - Total input paths to process : 1
2015-02-12 16:12:52,206 INFO  cascading.PlatformTestCase (PlatformTestCase.java:tearDown(209)) - copying to local /Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/test/output//hadoop2-mr1/assemblyhelpersplatform/countall
2015-02-12 16:12:52,218 INFO  util.HadoopUtil (HadoopUtil.java:findMainClass(336)) - using default application jar, may cause class not found exceptions on the cluster
2015-02-12 16:12:52,219 INFO  planner.HadoopPlanner (HadoopPlanner.java:initialize(225)) - using application jar: /Users/soberoi/.gradle/caches/modules-2/files-2.1/cascading/cascading-hadoop2-mr1/2.6.3/9ae016e2b8d246b9c1de9168e715d9f1b971aa4a/cascading-hadoop2-mr1-2.6.3.jar
2015-02-12 16:12:52,223 INFO  Configuration.deprecation (Configuration.java:warnOnceIfDeprecated(1022)) - mapred.used.genericoptionsparser is deprecated. Instead, use mapreduce.client.genericoptionsparser.used
2015-02-12 16:12:52,224 INFO  Configuration.deprecation (Configuration.java:warnOnceIfDeprecated(1022)) - mapred.job.tracker is deprecated. Instead, use mapreduce.jobtracker.address
2015-02-12 16:12:52,224 INFO  Configuration.deprecation (Configuration.java:warnOnceIfDeprecated(1022)) - mapred.output.compress is deprecated. Instead, use mapreduce.output.fileoutputformat.compress
2015-02-12 16:12:52,229 INFO  flow.Flow (BaseFlow.java:logInfo(1378)) - [name] starting
2015-02-12 16:12:52,229 INFO  flow.Flow (BaseFlow.java:logInfo(1378)) - [name]  source: Hfs["TextDelimited[['num', 'char']]"]["/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/classes/test/data/lhs.txt"]
2015-02-12 16:12:52,229 INFO  flow.Flow (BaseFlow.java:logInfo(1378)) - [name]  sink: Hfs["TextDelimited[['char', 'sum', 'count', 'average', 'average2', 'first']]"]["/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/test/output/hadoop2-mr1/assemblyhelpersplatform/multi"]
2015-02-12 16:12:52,229 INFO  flow.Flow (BaseFlow.java:logInfo(1378)) - [name]  parallel execution is enabled: false
2015-02-12 16:12:52,229 INFO  flow.Flow (BaseFlow.java:logInfo(1378)) - [name]  starting jobs: 1
2015-02-12 16:12:52,229 INFO  flow.Flow (BaseFlow.java:logInfo(1378)) - [name]  allocating threads: 1
2015-02-12 16:12:52,230 INFO  flow.FlowStep (BaseFlowStep.java:logInfo(834)) - [name] starting step: (1/1) ...mblyhelpersplatform/multi
2015-02-12 16:12:52,230 INFO  jvm.JvmMetrics (JvmMetrics.java:init(71)) - Cannot initialize JVM Metrics with processName=JobTracker, sessionId= - already initialized
2015-02-12 16:12:52,231 INFO  jvm.JvmMetrics (JvmMetrics.java:init(71)) - Cannot initialize JVM Metrics with processName=JobTracker, sessionId= - already initialized
2015-02-12 16:12:52,248 INFO  mapred.FileInputFormat (FileInputFormat.java:listStatus(247)) - Total input paths to process : 1
2015-02-12 16:12:52,265 INFO  mapreduce.JobSubmitter (JobSubmitter.java:submitJobInternal(492)) - number of splits:1
2015-02-12 16:12:52,277 INFO  mapreduce.JobSubmitter (JobSubmitter.java:printTokens(575)) - Submitting tokens for job: job_local1810252879_0026
2015-02-12 16:12:52,287 WARN  conf.Configuration (Configuration.java:loadProperty(2510)) - file:/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/tmp/cascading/staging/soberoi1810252879/.staging/job_local1810252879_0026/job.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.retry.interval;  Ignoring.
2015-02-12 16:12:52,287 WARN  conf.Configuration (Configuration.java:loadProperty(2510)) - file:/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/tmp/cascading/staging/soberoi1810252879/.staging/job_local1810252879_0026/job.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.attempts;  Ignoring.
2015-02-12 16:12:52,318 WARN  conf.Configuration (Configuration.java:loadProperty(2510)) - file:/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/test/tmp/mapred/local/localRunner/soberoi/job_local1810252879_0026/job_local1810252879_0026.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.retry.interval;  Ignoring.
2015-02-12 16:12:52,319 WARN  conf.Configuration (Configuration.java:loadProperty(2510)) - file:/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/test/tmp/mapred/local/localRunner/soberoi/job_local1810252879_0026/job_local1810252879_0026.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.attempts;  Ignoring.
2015-02-12 16:12:52,319 INFO  mapreduce.Job (Job.java:submit(1299)) - The url to track the job: http://localhost:8080/
2015-02-12 16:12:52,320 INFO  mapred.LocalJobRunner (LocalJobRunner.java:createOutputCommitter(471)) - OutputCommitter set in config null
2015-02-12 16:12:52,320 INFO  flow.FlowStep (BaseFlowStep.java:logInfo(834)) - [name] submitted hadoop job: job_local1810252879_0026
2015-02-12 16:12:52,320 INFO  mapred.LocalJobRunner (LocalJobRunner.java:createOutputCommitter(489)) - OutputCommitter is org.apache.hadoop.mapred.FileOutputCommitter
2015-02-12 16:12:52,320 INFO  flow.FlowStep (BaseFlowStep.java:logInfo(834)) - [name] tracking url: http://localhost:8080/
2015-02-12 16:12:52,322 INFO  mapred.LocalJobRunner (LocalJobRunner.java:runTasks(448)) - Waiting for map tasks
2015-02-12 16:12:52,322 INFO  mapred.LocalJobRunner (LocalJobRunner.java:run(224)) - Starting task: attempt_local1810252879_0026_m_000000_0
2015-02-12 16:12:52,323 INFO  util.ProcfsBasedProcessTree (ProcfsBasedProcessTree.java:isAvailable(181)) - ProcfsBasedProcessTree currently is supported only on Linux.
2015-02-12 16:12:52,323 INFO  mapred.Task (Task.java:initialize(587)) -  Using ResourceCalculatorProcessTree : null
2015-02-12 16:12:52,323 INFO  io.MultiInputSplit (MultiInputSplit.java:readFields(161)) - current split input path: file:/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/classes/test/data/lhs.txt
2015-02-12 16:12:52,323 INFO  mapred.MapTask (MapTask.java:updateJobWithSplit(482)) - Processing split: cascading.tap.hadoop.io.MultiInputSplit@66ce3c65
2015-02-12 16:12:52,324 INFO  mapred.MapTask (MapTask.java:runOldMapper(436)) - numReduceTasks: 1
2015-02-12 16:12:52,339 INFO  mapred.MapTask (MapTask.java:setEquator(1202)) - (EQUATOR) 0 kvi 26214396(104857584)
2015-02-12 16:12:52,340 INFO  mapred.MapTask (MapTask.java:init(995)) - mapreduce.task.io.sort.mb: 100
2015-02-12 16:12:52,340 INFO  mapred.MapTask (MapTask.java:init(996)) - soft limit at 83886080
2015-02-12 16:12:52,340 INFO  mapred.MapTask (MapTask.java:init(997)) - bufstart = 0; bufvoid = 104857600
2015-02-12 16:12:52,340 INFO  mapred.MapTask (MapTask.java:init(998)) - kvstart = 26214396; length = 6553600
2015-02-12 16:12:52,341 INFO  mapred.MapTask (MapTask.java:createSortingCollector(402)) - Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer
2015-02-12 16:12:52,341 INFO  hadoop.FlowMapper (FlowMapper.java:configure(71)) - cascading version: 2.6.3
2015-02-12 16:12:52,341 INFO  hadoop.FlowMapper (FlowMapper.java:configure(72)) - child jvm opts: -Xmx200m
2015-02-12 16:12:52,347 INFO  hadoop.FlowMapper (FlowMapper.java:configure(87)) - sourcing from: Hfs["TextDelimited[['num', 'char']]"]["/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/classes/test/data/lhs.txt"]
2015-02-12 16:12:52,348 INFO  hadoop.FlowMapper (FlowMapper.java:configure(90)) - sinking to: GroupBy(name)[by:[{1}:'char']]
2015-02-12 16:12:52,348 INFO  assembly.AggregateBy (AggregateBy.java:prepare(267)) - using threshold value: 2
2015-02-12 16:12:52,349 INFO  assembly.AggregateBy (AggregateBy.java:removeEldestEntry(327)) - flushed keys num times: 1, with threshold: 2
2015-02-12 16:12:52,349 INFO  assembly.AggregateBy (AggregateBy.java:removeEldestEntry(328)) - mem on flush (mb), free: 196, total: 512, max: 672
2015-02-12 16:12:52,350 INFO  assembly.AggregateBy (AggregateBy.java:removeEldestEntry(333)) - total mem is 76% of max mem, to better utilize unused memory consider increasing current LRU threshold with system property "cascading.aggregateby.threshold"
2015-02-12 16:12:52,350 INFO  assembly.AggregateBy (AggregateBy.java:removeEldestEntry(327)) - flushed keys num times: 3, with threshold: 2
2015-02-12 16:12:52,350 INFO  assembly.AggregateBy (AggregateBy.java:removeEldestEntry(328)) - mem on flush (mb), free: 196, total: 512, max: 672
2015-02-12 16:12:52,351 INFO  assembly.AggregateBy (AggregateBy.java:removeEldestEntry(333)) - total mem is 76% of max mem, to better utilize unused memory consider increasing current LRU threshold with system property "cascading.aggregateby.threshold"
2015-02-12 16:12:52,351 INFO  assembly.AggregateBy (AggregateBy.java:removeEldestEntry(327)) - flushed keys num times: 5, with threshold: 2
2015-02-12 16:12:52,351 INFO  assembly.AggregateBy (AggregateBy.java:removeEldestEntry(328)) - mem on flush (mb), free: 196, total: 512, max: 672
2015-02-12 16:12:52,351 INFO  assembly.AggregateBy (AggregateBy.java:removeEldestEntry(333)) - total mem is 76% of max mem, to better utilize unused memory consider increasing current LRU threshold with system property "cascading.aggregateby.threshold"
2015-02-12 16:12:52,352 INFO  assembly.AggregateBy (AggregateBy.java:removeEldestEntry(327)) - flushed keys num times: 7, with threshold: 2
2015-02-12 16:12:52,352 INFO  assembly.AggregateBy (AggregateBy.java:removeEldestEntry(328)) - mem on flush (mb), free: 196, total: 512, max: 672
2015-02-12 16:12:52,352 INFO  assembly.AggregateBy (AggregateBy.java:removeEldestEntry(333)) - total mem is 76% of max mem, to better utilize unused memory consider increasing current LRU threshold with system property "cascading.aggregateby.threshold"
2015-02-12 16:12:52,353 INFO  mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) - 
2015-02-12 16:12:52,353 INFO  mapred.MapTask (MapTask.java:flush(1457)) - Starting flush of map output
2015-02-12 16:12:52,353 INFO  mapred.MapTask (MapTask.java:flush(1475)) - Spilling map output
2015-02-12 16:12:52,353 INFO  mapred.MapTask (MapTask.java:flush(1476)) - bufstart = 0; bufend = 432; bufvoid = 104857600
2015-02-12 16:12:52,354 INFO  mapred.MapTask (MapTask.java:flush(1478)) - kvstart = 26214396(104857584); kvend = 26214364(104857456); length = 33/6553600
2015-02-12 16:12:52,356 INFO  mapred.MapTask (MapTask.java:sortAndSpill(1660)) - Finished spill 0
2015-02-12 16:12:52,358 INFO  mapred.Task (Task.java:done(1001)) - Task:attempt_local1810252879_0026_m_000000_0 is done. And is in the process of committing
2015-02-12 16:12:52,359 INFO  mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) - file:/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/classes/test/data/lhs.txt:0+52
2015-02-12 16:12:52,359 INFO  mapred.Task (Task.java:sendDone(1121)) - Task 'attempt_local1810252879_0026_m_000000_0' done.
2015-02-12 16:12:52,360 INFO  mapred.LocalJobRunner (LocalJobRunner.java:run(249)) - Finishing task: attempt_local1810252879_0026_m_000000_0
2015-02-12 16:12:52,360 INFO  mapred.LocalJobRunner (LocalJobRunner.java:runTasks(456)) - map task executor complete.
2015-02-12 16:12:52,360 INFO  mapred.LocalJobRunner (LocalJobRunner.java:runTasks(448)) - Waiting for reduce tasks
2015-02-12 16:12:52,360 INFO  mapred.LocalJobRunner (LocalJobRunner.java:run(302)) - Starting task: attempt_local1810252879_0026_r_000000_0
2015-02-12 16:12:52,362 INFO  util.ProcfsBasedProcessTree (ProcfsBasedProcessTree.java:isAvailable(181)) - ProcfsBasedProcessTree currently is supported only on Linux.
2015-02-12 16:12:52,362 INFO  mapred.Task (Task.java:initialize(587)) -  Using ResourceCalculatorProcessTree : null
2015-02-12 16:12:52,362 INFO  mapred.ReduceTask (ReduceTask.java:run(362)) - Using ShuffleConsumerPlugin: org.apache.hadoop.mapreduce.task.reduce.Shuffle@3ecf7dc2
2015-02-12 16:12:52,362 INFO  reduce.MergeManagerImpl (MergeManagerImpl.java:<init>(195)) - MergerManager: memoryLimit=493250144, maxSingleShuffleLimit=123312536, mergeThreshold=325545120, ioSortFactor=10, memToMemMergeOutputsThreshold=10
2015-02-12 16:12:52,363 INFO  reduce.EventFetcher (EventFetcher.java:run(61)) - attempt_local1810252879_0026_r_000000_0 Thread started: EventFetcher for fetching Map Completion Events
2015-02-12 16:12:52,364 INFO  reduce.LocalFetcher (LocalFetcher.java:copyMapOutput(141)) - localfetcher#22 about to shuffle output of map attempt_local1810252879_0026_m_000000_0 decomp: 452 len: 456 to MEMORY
2015-02-12 16:12:52,364 INFO  reduce.InMemoryMapOutput (InMemoryMapOutput.java:shuffle(100)) - Read 452 bytes from map-output for attempt_local1810252879_0026_m_000000_0
2015-02-12 16:12:52,364 INFO  reduce.MergeManagerImpl (MergeManagerImpl.java:closeInMemoryFile(313)) - closeInMemoryFile -> map-output of size: 452, inMemoryMapOutputs.size() -> 1, commitMemory -> 0, usedMemory ->452
2015-02-12 16:12:52,365 INFO  reduce.EventFetcher (EventFetcher.java:run(76)) - EventFetcher is interrupted.. Returning
2015-02-12 16:12:52,365 INFO  mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) - 1 / 1 copied.
2015-02-12 16:12:52,366 INFO  reduce.MergeManagerImpl (MergeManagerImpl.java:finalMerge(673)) - finalMerge called with 1 in-memory map-outputs and 0 on-disk map-outputs
2015-02-12 16:12:52,367 INFO  mapred.Merger (Merger.java:merge(606)) - Merging 1 sorted segments
2015-02-12 16:12:52,368 INFO  mapred.Merger (Merger.java:merge(705)) - Down to the last merge-pass, with 1 segments left of total size: 436 bytes
2015-02-12 16:12:52,369 INFO  reduce.MergeManagerImpl (MergeManagerImpl.java:finalMerge(750)) - Merged 1 segments, 452 bytes to disk to satisfy reduce memory limit
2015-02-12 16:12:52,369 INFO  reduce.MergeManagerImpl (MergeManagerImpl.java:finalMerge(780)) - Merging 1 files, 456 bytes from disk
2015-02-12 16:12:52,369 INFO  reduce.MergeManagerImpl (MergeManagerImpl.java:finalMerge(795)) - Merging 0 segments, 0 bytes from memory into reduce
2015-02-12 16:12:52,369 INFO  mapred.Merger (Merger.java:merge(606)) - Merging 1 sorted segments
2015-02-12 16:12:52,370 INFO  mapred.Merger (Merger.java:merge(705)) - Down to the last merge-pass, with 1 segments left of total size: 436 bytes
2015-02-12 16:12:52,370 INFO  mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) - 1 / 1 copied.
2015-02-12 16:12:52,370 INFO  hadoop.FlowReducer (FlowReducer.java:configure(78)) - cascading version: 2.6.3
2015-02-12 16:12:52,370 INFO  hadoop.FlowReducer (FlowReducer.java:configure(79)) - child jvm opts: -Xmx200m
2015-02-12 16:12:52,377 INFO  hadoop.FlowReducer (FlowReducer.java:configure(97)) - sourcing from: GroupBy(name)[by:[{1}:'char']]
2015-02-12 16:12:52,377 INFO  hadoop.FlowReducer (FlowReducer.java:configure(100)) - sinking to: Hfs["TextDelimited[['char', 'sum', 'count', 'average', 'average2', 'first']]"]["/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/test/output/hadoop2-mr1/assemblyhelpersplatform/multi"]
2015-02-12 16:12:52,385 INFO  mapred.Task (Task.java:done(1001)) - Task:attempt_local1810252879_0026_r_000000_0 is done. And is in the process of committing
2015-02-12 16:12:52,386 INFO  mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) - 1 / 1 copied.
2015-02-12 16:12:52,386 INFO  mapred.Task (Task.java:commit(1162)) - Task attempt_local1810252879_0026_r_000000_0 is allowed to commit now
2015-02-12 16:12:52,386 INFO  output.FileOutputCommitter (FileOutputCommitter.java:commitTask(439)) - Saved output of task 'attempt_local1810252879_0026_r_000000_0' to file:/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/test/output/hadoop2-mr1/assemblyhelpersplatform/multi/_temporary/0/task_local1810252879_0026_r_000000
2015-02-12 16:12:52,387 INFO  mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) - reduce > reduce
2015-02-12 16:12:52,387 INFO  mapred.Task (Task.java:sendDone(1121)) - Task 'attempt_local1810252879_0026_r_000000_0' done.
2015-02-12 16:12:52,387 INFO  mapred.LocalJobRunner (LocalJobRunner.java:run(325)) - Finishing task: attempt_local1810252879_0026_r_000000_0
2015-02-12 16:12:52,387 INFO  mapred.LocalJobRunner (LocalJobRunner.java:runTasks(456)) - reduce task executor complete.
2015-02-12 16:12:52,404 INFO  util.Hadoop18TapUtil (Hadoop18TapUtil.java:cleanTempPath(219)) - deleting temp path /Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/test/output/hadoop2-mr1/assemblyhelpersplatform/multi/_temporary
2015-02-12 16:12:52,423 INFO  mapred.FileInputFormat (FileInputFormat.java:listStatus(247)) - Total input paths to process : 1
2015-02-12 16:12:52,445 INFO  mapred.FileInputFormat (FileInputFormat.java:listStatus(247)) - Total input paths to process : 1
2015-02-12 16:12:52,446 INFO  cascading.PlatformTestCase (PlatformTestCase.java:tearDown(209)) - copying to local /Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/test/output//hadoop2-mr1/assemblyhelpersplatform/multi
2015-02-12 16:12:52,467 INFO  util.HadoopUtil (HadoopUtil.java:findMainClass(336)) - using default application jar, may cause class not found exceptions on the cluster
2015-02-12 16:12:52,467 INFO  planner.HadoopPlanner (HadoopPlanner.java:initialize(225)) - using application jar: /Users/soberoi/.gradle/caches/modules-2/files-2.1/cascading/cascading-hadoop2-mr1/2.6.3/9ae016e2b8d246b9c1de9168e715d9f1b971aa4a/cascading-hadoop2-mr1-2.6.3.jar
2015-02-12 16:12:52,470 INFO  Configuration.deprecation (Configuration.java:warnOnceIfDeprecated(1022)) - mapred.used.genericoptionsparser is deprecated. Instead, use mapreduce.client.genericoptionsparser.used
2015-02-12 16:12:52,471 INFO  Configuration.deprecation (Configuration.java:warnOnceIfDeprecated(1022)) - mapred.job.tracker is deprecated. Instead, use mapreduce.jobtracker.address
2015-02-12 16:12:52,472 INFO  Configuration.deprecation (Configuration.java:warnOnceIfDeprecated(1022)) - mapred.output.compress is deprecated. Instead, use mapreduce.output.fileoutputformat.compress
2015-02-12 16:12:52,475 INFO  flow.Flow (BaseFlow.java:logInfo(1378)) - [shape] starting
2015-02-12 16:12:52,475 INFO  flow.Flow (BaseFlow.java:logInfo(1378)) - [shape]  source: Hfs["TextLine[['offset', 'line']->[ALL]]"]["/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/classes/test/data/lower.txt"]
2015-02-12 16:12:52,475 INFO  flow.Flow (BaseFlow.java:logInfo(1378)) - [shape]  sink: Hfs["TextLine[['num']]"]["/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/test/output/hadoop2-mr1/assemblyhelpersplatform/retainnarrow"]
2015-02-12 16:12:52,475 INFO  flow.Flow (BaseFlow.java:logInfo(1378)) - [shape]  parallel execution is enabled: false
2015-02-12 16:12:52,475 INFO  flow.Flow (BaseFlow.java:logInfo(1378)) - [shape]  starting jobs: 1
2015-02-12 16:12:52,475 INFO  flow.Flow (BaseFlow.java:logInfo(1378)) - [shape]  allocating threads: 1
2015-02-12 16:12:52,476 INFO  flow.FlowStep (BaseFlowStep.java:logInfo(834)) - [shape] starting step: (1/1) ...persplatform/retainnarrow
2015-02-12 16:12:52,476 INFO  jvm.JvmMetrics (JvmMetrics.java:init(71)) - Cannot initialize JVM Metrics with processName=JobTracker, sessionId= - already initialized
2015-02-12 16:12:52,477 INFO  jvm.JvmMetrics (JvmMetrics.java:init(71)) - Cannot initialize JVM Metrics with processName=JobTracker, sessionId= - already initialized
2015-02-12 16:12:52,494 INFO  mapred.FileInputFormat (FileInputFormat.java:listStatus(247)) - Total input paths to process : 1
2015-02-12 16:12:52,510 INFO  mapreduce.JobSubmitter (JobSubmitter.java:submitJobInternal(492)) - number of splits:1
2015-02-12 16:12:52,521 INFO  mapreduce.JobSubmitter (JobSubmitter.java:printTokens(575)) - Submitting tokens for job: job_local783720211_0027
2015-02-12 16:12:52,532 WARN  conf.Configuration (Configuration.java:loadProperty(2510)) - file:/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/tmp/cascading/staging/soberoi783720211/.staging/job_local783720211_0027/job.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.retry.interval;  Ignoring.
2015-02-12 16:12:52,533 WARN  conf.Configuration (Configuration.java:loadProperty(2510)) - file:/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/tmp/cascading/staging/soberoi783720211/.staging/job_local783720211_0027/job.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.attempts;  Ignoring.
2015-02-12 16:12:52,560 WARN  conf.Configuration (Configuration.java:loadProperty(2510)) - file:/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/test/tmp/mapred/local/localRunner/soberoi/job_local783720211_0027/job_local783720211_0027.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.retry.interval;  Ignoring.
2015-02-12 16:12:52,560 WARN  conf.Configuration (Configuration.java:loadProperty(2510)) - file:/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/test/tmp/mapred/local/localRunner/soberoi/job_local783720211_0027/job_local783720211_0027.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.attempts;  Ignoring.
2015-02-12 16:12:52,561 INFO  mapreduce.Job (Job.java:submit(1299)) - The url to track the job: http://localhost:8080/
2015-02-12 16:12:52,561 INFO  mapred.LocalJobRunner (LocalJobRunner.java:createOutputCommitter(471)) - OutputCommitter set in config null
2015-02-12 16:12:52,561 INFO  flow.FlowStep (BaseFlowStep.java:logInfo(834)) - [shape] submitted hadoop job: job_local783720211_0027
2015-02-12 16:12:52,561 INFO  mapred.LocalJobRunner (LocalJobRunner.java:createOutputCommitter(489)) - OutputCommitter is org.apache.hadoop.mapred.FileOutputCommitter
2015-02-12 16:12:52,562 INFO  flow.FlowStep (BaseFlowStep.java:logInfo(834)) - [shape] tracking url: http://localhost:8080/
2015-02-12 16:12:52,563 INFO  mapred.LocalJobRunner (LocalJobRunner.java:runTasks(448)) - Waiting for map tasks
2015-02-12 16:12:52,563 INFO  mapred.LocalJobRunner (LocalJobRunner.java:run(224)) - Starting task: attempt_local783720211_0027_m_000000_0
2015-02-12 16:12:52,565 INFO  util.ProcfsBasedProcessTree (ProcfsBasedProcessTree.java:isAvailable(181)) - ProcfsBasedProcessTree currently is supported only on Linux.
2015-02-12 16:12:52,565 INFO  mapred.Task (Task.java:initialize(587)) -  Using ResourceCalculatorProcessTree : null
2015-02-12 16:12:52,566 INFO  io.MultiInputSplit (MultiInputSplit.java:readFields(161)) - current split input path: file:/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/classes/test/data/lower.txt
2015-02-12 16:12:52,566 INFO  mapred.MapTask (MapTask.java:updateJobWithSplit(482)) - Processing split: cascading.tap.hadoop.io.MultiInputSplit@2fe6f732
2015-02-12 16:12:52,566 INFO  mapred.MapTask (MapTask.java:runOldMapper(436)) - numReduceTasks: 0
2015-02-12 16:12:52,572 INFO  hadoop.FlowMapper (FlowMapper.java:configure(71)) - cascading version: 2.6.3
2015-02-12 16:12:52,572 INFO  hadoop.FlowMapper (FlowMapper.java:configure(72)) - child jvm opts: -Xmx200m
2015-02-12 16:12:52,574 INFO  hadoop.FlowMapper (FlowMapper.java:configure(87)) - sourcing from: Hfs["TextLine[['offset', 'line']->[ALL]]"]["/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/classes/test/data/lower.txt"]
2015-02-12 16:12:52,575 INFO  hadoop.FlowMapper (FlowMapper.java:configure(90)) - sinking to: Hfs["TextLine[['num']]"]["/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/test/output/hadoop2-mr1/assemblyhelpersplatform/retainnarrow"]
2015-02-12 16:12:52,576 INFO  mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) - 
2015-02-12 16:12:52,577 INFO  mapred.Task (Task.java:done(1001)) - Task:attempt_local783720211_0027_m_000000_0 is done. And is in the process of committing
2015-02-12 16:12:52,578 INFO  mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) - 
2015-02-12 16:12:52,578 INFO  mapred.Task (Task.java:commit(1162)) - Task attempt_local783720211_0027_m_000000_0 is allowed to commit now
2015-02-12 16:12:52,578 INFO  output.FileOutputCommitter (FileOutputCommitter.java:commitTask(439)) - Saved output of task 'attempt_local783720211_0027_m_000000_0' to file:/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/test/output/hadoop2-mr1/assemblyhelpersplatform/retainnarrow/_temporary/0/task_local783720211_0027_m_000000
2015-02-12 16:12:52,579 INFO  mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) - file:/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/classes/test/data/lower.txt:0+19
2015-02-12 16:12:52,579 INFO  mapred.Task (Task.java:sendDone(1121)) - Task 'attempt_local783720211_0027_m_000000_0' done.
2015-02-12 16:12:52,579 INFO  mapred.LocalJobRunner (LocalJobRunner.java:run(249)) - Finishing task: attempt_local783720211_0027_m_000000_0
2015-02-12 16:12:52,579 INFO  mapred.LocalJobRunner (LocalJobRunner.java:runTasks(456)) - map task executor complete.
2015-02-12 16:12:52,603 INFO  util.Hadoop18TapUtil (Hadoop18TapUtil.java:cleanTempPath(219)) - deleting temp path /Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/test/output/hadoop2-mr1/assemblyhelpersplatform/retainnarrow/_temporary
2015-02-12 16:12:52,624 INFO  mapred.FileInputFormat (FileInputFormat.java:listStatus(247)) - Total input paths to process : 1
2015-02-12 16:12:52,625 INFO  cascading.PlatformTestCase (PlatformTestCase.java:tearDown(209)) - copying to local /Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/test/output//hadoop2-mr1/assemblyhelpersplatform/retainnarrow
2015-02-12 16:12:52,636 INFO  util.HadoopUtil (HadoopUtil.java:findMainClass(336)) - using default application jar, may cause class not found exceptions on the cluster
2015-02-12 16:12:52,636 INFO  planner.HadoopPlanner (HadoopPlanner.java:initialize(225)) - using application jar: /Users/soberoi/.gradle/caches/modules-2/files-2.1/cascading/cascading-hadoop2-mr1/2.6.3/9ae016e2b8d246b9c1de9168e715d9f1b971aa4a/cascading-hadoop2-mr1-2.6.3.jar
2015-02-12 16:12:52,639 INFO  Configuration.deprecation (Configuration.java:warnOnceIfDeprecated(1022)) - mapred.used.genericoptionsparser is deprecated. Instead, use mapreduce.client.genericoptionsparser.used
2015-02-12 16:12:52,640 INFO  Configuration.deprecation (Configuration.java:warnOnceIfDeprecated(1022)) - mapred.job.tracker is deprecated. Instead, use mapreduce.jobtracker.address
2015-02-12 16:12:52,640 INFO  Configuration.deprecation (Configuration.java:warnOnceIfDeprecated(1022)) - mapred.output.compress is deprecated. Instead, use mapreduce.output.fileoutputformat.compress
2015-02-12 16:12:52,643 INFO  flow.Flow (BaseFlow.java:logInfo(1378)) - [shape] starting
2015-02-12 16:12:52,643 INFO  flow.Flow (BaseFlow.java:logInfo(1378)) - [shape]  source: Hfs["TextLine[['offset', 'line']->[ALL]]"]["/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/classes/test/data/lower.txt"]
2015-02-12 16:12:52,643 INFO  flow.Flow (BaseFlow.java:logInfo(1378)) - [shape]  sink: Hfs["TextLine[['item']->['char', 'item']]"]["/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/test/output/hadoop2-mr1/assemblyhelpersplatform/renamenarrow"]
2015-02-12 16:12:52,643 INFO  flow.Flow (BaseFlow.java:logInfo(1378)) - [shape]  parallel execution is enabled: false
2015-02-12 16:12:52,644 INFO  flow.Flow (BaseFlow.java:logInfo(1378)) - [shape]  starting jobs: 1
2015-02-12 16:12:52,644 INFO  flow.Flow (BaseFlow.java:logInfo(1378)) - [shape]  allocating threads: 1
2015-02-12 16:12:52,644 INFO  flow.FlowStep (BaseFlowStep.java:logInfo(834)) - [shape] starting step: (1/1) ...persplatform/renamenarrow
2015-02-12 16:12:52,645 INFO  jvm.JvmMetrics (JvmMetrics.java:init(71)) - Cannot initialize JVM Metrics with processName=JobTracker, sessionId= - already initialized
2015-02-12 16:12:52,645 INFO  jvm.JvmMetrics (JvmMetrics.java:init(71)) - Cannot initialize JVM Metrics with processName=JobTracker, sessionId= - already initialized
2015-02-12 16:12:52,663 INFO  mapred.FileInputFormat (FileInputFormat.java:listStatus(247)) - Total input paths to process : 1
2015-02-12 16:12:52,680 INFO  mapreduce.JobSubmitter (JobSubmitter.java:submitJobInternal(492)) - number of splits:1
2015-02-12 16:12:52,691 INFO  mapreduce.JobSubmitter (JobSubmitter.java:printTokens(575)) - Submitting tokens for job: job_local1949352354_0028
2015-02-12 16:12:52,702 WARN  conf.Configuration (Configuration.java:loadProperty(2510)) - file:/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/tmp/cascading/staging/soberoi1949352354/.staging/job_local1949352354_0028/job.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.retry.interval;  Ignoring.
2015-02-12 16:12:52,703 WARN  conf.Configuration (Configuration.java:loadProperty(2510)) - file:/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/tmp/cascading/staging/soberoi1949352354/.staging/job_local1949352354_0028/job.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.attempts;  Ignoring.
2015-02-12 16:12:52,730 WARN  conf.Configuration (Configuration.java:loadProperty(2510)) - file:/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/test/tmp/mapred/local/localRunner/soberoi/job_local1949352354_0028/job_local1949352354_0028.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.retry.interval;  Ignoring.
2015-02-12 16:12:52,731 WARN  conf.Configuration (Configuration.java:loadProperty(2510)) - file:/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/test/tmp/mapred/local/localRunner/soberoi/job_local1949352354_0028/job_local1949352354_0028.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.attempts;  Ignoring.
2015-02-12 16:12:52,732 INFO  mapreduce.Job (Job.java:submit(1299)) - The url to track the job: http://localhost:8080/
2015-02-12 16:12:52,732 INFO  mapred.LocalJobRunner (LocalJobRunner.java:createOutputCommitter(471)) - OutputCommitter set in config null
2015-02-12 16:12:52,732 INFO  flow.FlowStep (BaseFlowStep.java:logInfo(834)) - [shape] submitted hadoop job: job_local1949352354_0028
2015-02-12 16:12:52,732 INFO  mapred.LocalJobRunner (LocalJobRunner.java:createOutputCommitter(489)) - OutputCommitter is org.apache.hadoop.mapred.FileOutputCommitter
2015-02-12 16:12:52,732 INFO  flow.FlowStep (BaseFlowStep.java:logInfo(834)) - [shape] tracking url: http://localhost:8080/
2015-02-12 16:12:52,734 INFO  mapred.LocalJobRunner (LocalJobRunner.java:runTasks(448)) - Waiting for map tasks
2015-02-12 16:12:52,734 INFO  mapred.LocalJobRunner (LocalJobRunner.java:run(224)) - Starting task: attempt_local1949352354_0028_m_000000_0
2015-02-12 16:12:52,735 INFO  util.ProcfsBasedProcessTree (ProcfsBasedProcessTree.java:isAvailable(181)) - ProcfsBasedProcessTree currently is supported only on Linux.
2015-02-12 16:12:52,735 INFO  mapred.Task (Task.java:initialize(587)) -  Using ResourceCalculatorProcessTree : null
2015-02-12 16:12:52,736 INFO  io.MultiInputSplit (MultiInputSplit.java:readFields(161)) - current split input path: file:/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/classes/test/data/lower.txt
2015-02-12 16:12:52,736 INFO  mapred.MapTask (MapTask.java:updateJobWithSplit(482)) - Processing split: cascading.tap.hadoop.io.MultiInputSplit@2d3c2ba1
2015-02-12 16:12:52,736 INFO  mapred.MapTask (MapTask.java:runOldMapper(436)) - numReduceTasks: 0
2015-02-12 16:12:52,741 INFO  hadoop.FlowMapper (FlowMapper.java:configure(71)) - cascading version: 2.6.3
2015-02-12 16:12:52,742 INFO  hadoop.FlowMapper (FlowMapper.java:configure(72)) - child jvm opts: -Xmx200m
2015-02-12 16:12:52,744 INFO  hadoop.FlowMapper (FlowMapper.java:configure(87)) - sourcing from: Hfs["TextLine[['offset', 'line']->[ALL]]"]["/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/classes/test/data/lower.txt"]
2015-02-12 16:12:52,744 INFO  hadoop.FlowMapper (FlowMapper.java:configure(90)) - sinking to: Hfs["TextLine[['item']->['char', 'item']]"]["/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/test/output/hadoop2-mr1/assemblyhelpersplatform/renamenarrow"]
2015-02-12 16:12:52,746 INFO  mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) - 
2015-02-12 16:12:52,746 INFO  mapred.Task (Task.java:done(1001)) - Task:attempt_local1949352354_0028_m_000000_0 is done. And is in the process of committing
2015-02-12 16:12:52,747 INFO  mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) - 
2015-02-12 16:12:52,748 INFO  mapred.Task (Task.java:commit(1162)) - Task attempt_local1949352354_0028_m_000000_0 is allowed to commit now
2015-02-12 16:12:52,748 INFO  output.FileOutputCommitter (FileOutputCommitter.java:commitTask(439)) - Saved output of task 'attempt_local1949352354_0028_m_000000_0' to file:/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/test/output/hadoop2-mr1/assemblyhelpersplatform/renamenarrow/_temporary/0/task_local1949352354_0028_m_000000
2015-02-12 16:12:52,749 INFO  mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) - file:/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/classes/test/data/lower.txt:0+19
2015-02-12 16:12:52,749 INFO  mapred.Task (Task.java:sendDone(1121)) - Task 'attempt_local1949352354_0028_m_000000_0' done.
2015-02-12 16:12:52,749 INFO  mapred.LocalJobRunner (LocalJobRunner.java:run(249)) - Finishing task: attempt_local1949352354_0028_m_000000_0
2015-02-12 16:12:52,749 INFO  mapred.LocalJobRunner (LocalJobRunner.java:runTasks(456)) - map task executor complete.
2015-02-12 16:12:52,758 INFO  util.Hadoop18TapUtil (Hadoop18TapUtil.java:cleanTempPath(219)) - deleting temp path /Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/test/output/hadoop2-mr1/assemblyhelpersplatform/renamenarrow/_temporary
2015-02-12 16:12:52,779 INFO  mapred.FileInputFormat (FileInputFormat.java:listStatus(247)) - Total input paths to process : 1
2015-02-12 16:12:52,780 INFO  cascading.PlatformTestCase (PlatformTestCase.java:tearDown(209)) - copying to local /Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/test/output//hadoop2-mr1/assemblyhelpersplatform/renamenarrow
2015-02-12 16:12:52,792 INFO  util.HadoopUtil (HadoopUtil.java:findMainClass(336)) - using default application jar, may cause class not found exceptions on the cluster
2015-02-12 16:12:52,792 INFO  planner.HadoopPlanner (HadoopPlanner.java:initialize(225)) - using application jar: /Users/soberoi/.gradle/caches/modules-2/files-2.1/cascading/cascading-hadoop2-mr1/2.6.3/9ae016e2b8d246b9c1de9168e715d9f1b971aa4a/cascading-hadoop2-mr1-2.6.3.jar
2015-02-12 16:12:52,795 INFO  Configuration.deprecation (Configuration.java:warnOnceIfDeprecated(1022)) - mapred.used.genericoptionsparser is deprecated. Instead, use mapreduce.client.genericoptionsparser.used
2015-02-12 16:12:52,796 INFO  Configuration.deprecation (Configuration.java:warnOnceIfDeprecated(1022)) - mapred.job.tracker is deprecated. Instead, use mapreduce.jobtracker.address
2015-02-12 16:12:52,797 INFO  Configuration.deprecation (Configuration.java:warnOnceIfDeprecated(1022)) - mapred.output.compress is deprecated. Instead, use mapreduce.output.fileoutputformat.compress
2015-02-12 16:12:52,800 INFO  flow.Flow (BaseFlow.java:logInfo(1378)) - [average] starting
2015-02-12 16:12:52,800 INFO  flow.Flow (BaseFlow.java:logInfo(1378)) - [average]  source: Hfs["TextDelimited[['num', 'char']]"]["/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/classes/test/data/lhs.txt"]
2015-02-12 16:12:52,800 INFO  flow.Flow (BaseFlow.java:logInfo(1378)) - [average]  sink: Hfs["TextDelimited[['char', 'average']]"]["/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/test/output/hadoop2-mr1/assemblyhelpersplatform/averagenull"]
2015-02-12 16:12:52,800 INFO  flow.Flow (BaseFlow.java:logInfo(1378)) - [average]  parallel execution is enabled: false
2015-02-12 16:12:52,801 INFO  flow.Flow (BaseFlow.java:logInfo(1378)) - [average]  starting jobs: 1
2015-02-12 16:12:52,801 INFO  flow.Flow (BaseFlow.java:logInfo(1378)) - [average]  allocating threads: 1
2015-02-12 16:12:52,801 INFO  flow.FlowStep (BaseFlowStep.java:logInfo(834)) - [average] starting step: (1/1) ...lpersplatform/averagenull
2015-02-12 16:12:52,801 INFO  jvm.JvmMetrics (JvmMetrics.java:init(71)) - Cannot initialize JVM Metrics with processName=JobTracker, sessionId= - already initialized
2015-02-12 16:12:52,802 INFO  jvm.JvmMetrics (JvmMetrics.java:init(71)) - Cannot initialize JVM Metrics with processName=JobTracker, sessionId= - already initialized
2015-02-12 16:12:52,820 INFO  mapred.FileInputFormat (FileInputFormat.java:listStatus(247)) - Total input paths to process : 1
2015-02-12 16:12:52,836 INFO  mapreduce.JobSubmitter (JobSubmitter.java:submitJobInternal(492)) - number of splits:1
2015-02-12 16:12:52,852 INFO  mapreduce.JobSubmitter (JobSubmitter.java:printTokens(575)) - Submitting tokens for job: job_local1666493537_0029
2015-02-12 16:12:52,863 WARN  conf.Configuration (Configuration.java:loadProperty(2510)) - file:/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/tmp/cascading/staging/soberoi1666493537/.staging/job_local1666493537_0029/job.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.retry.interval;  Ignoring.
2015-02-12 16:12:52,864 WARN  conf.Configuration (Configuration.java:loadProperty(2510)) - file:/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/tmp/cascading/staging/soberoi1666493537/.staging/job_local1666493537_0029/job.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.attempts;  Ignoring.
2015-02-12 16:12:52,892 WARN  conf.Configuration (Configuration.java:loadProperty(2510)) - file:/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/test/tmp/mapred/local/localRunner/soberoi/job_local1666493537_0029/job_local1666493537_0029.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.retry.interval;  Ignoring.
2015-02-12 16:12:52,893 WARN  conf.Configuration (Configuration.java:loadProperty(2510)) - file:/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/test/tmp/mapred/local/localRunner/soberoi/job_local1666493537_0029/job_local1666493537_0029.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.attempts;  Ignoring.
2015-02-12 16:12:52,894 INFO  mapreduce.Job (Job.java:submit(1299)) - The url to track the job: http://localhost:8080/
2015-02-12 16:12:52,894 INFO  mapred.LocalJobRunner (LocalJobRunner.java:createOutputCommitter(471)) - OutputCommitter set in config null
2015-02-12 16:12:52,894 INFO  flow.FlowStep (BaseFlowStep.java:logInfo(834)) - [average] submitted hadoop job: job_local1666493537_0029
2015-02-12 16:12:52,894 INFO  mapred.LocalJobRunner (LocalJobRunner.java:createOutputCommitter(489)) - OutputCommitter is org.apache.hadoop.mapred.FileOutputCommitter
2015-02-12 16:12:52,894 INFO  flow.FlowStep (BaseFlowStep.java:logInfo(834)) - [average] tracking url: http://localhost:8080/
2015-02-12 16:12:52,896 INFO  mapred.LocalJobRunner (LocalJobRunner.java:runTasks(448)) - Waiting for map tasks
2015-02-12 16:12:52,896 INFO  mapred.LocalJobRunner (LocalJobRunner.java:run(224)) - Starting task: attempt_local1666493537_0029_m_000000_0
2015-02-12 16:12:52,897 INFO  util.ProcfsBasedProcessTree (ProcfsBasedProcessTree.java:isAvailable(181)) - ProcfsBasedProcessTree currently is supported only on Linux.
2015-02-12 16:12:52,897 INFO  mapred.Task (Task.java:initialize(587)) -  Using ResourceCalculatorProcessTree : null
2015-02-12 16:12:52,898 INFO  io.MultiInputSplit (MultiInputSplit.java:readFields(161)) - current split input path: file:/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/classes/test/data/lhs.txt
2015-02-12 16:12:52,898 INFO  mapred.MapTask (MapTask.java:updateJobWithSplit(482)) - Processing split: cascading.tap.hadoop.io.MultiInputSplit@1d4a0efb
2015-02-12 16:12:52,898 INFO  mapred.MapTask (MapTask.java:runOldMapper(436)) - numReduceTasks: 1
2015-02-12 16:12:52,909 INFO  mapred.MapTask (MapTask.java:setEquator(1202)) - (EQUATOR) 0 kvi 26214396(104857584)
2015-02-12 16:12:52,910 INFO  mapred.MapTask (MapTask.java:init(995)) - mapreduce.task.io.sort.mb: 100
2015-02-12 16:12:52,910 INFO  mapred.MapTask (MapTask.java:init(996)) - soft limit at 83886080
2015-02-12 16:12:52,910 INFO  mapred.MapTask (MapTask.java:init(997)) - bufstart = 0; bufvoid = 104857600
2015-02-12 16:12:52,910 INFO  mapred.MapTask (MapTask.java:init(998)) - kvstart = 26214396; length = 6553600
2015-02-12 16:12:52,911 INFO  mapred.MapTask (MapTask.java:createSortingCollector(402)) - Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer
2015-02-12 16:12:52,911 INFO  hadoop.FlowMapper (FlowMapper.java:configure(71)) - cascading version: 2.6.3
2015-02-12 16:12:52,911 INFO  hadoop.FlowMapper (FlowMapper.java:configure(72)) - child jvm opts: -Xmx200m
2015-02-12 16:12:52,914 INFO  hadoop.FlowMapper (FlowMapper.java:configure(87)) - sourcing from: Hfs["TextDelimited[['num', 'char']]"]["/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/classes/test/data/lhs.txt"]
2015-02-12 16:12:52,914 INFO  hadoop.FlowMapper (FlowMapper.java:configure(90)) - sinking to: GroupBy(average)[by:[{1}:'char']]
2015-02-12 16:12:52,914 INFO  assembly.AggregateBy (AggregateBy.java:prepare(267)) - using threshold value: 2
2015-02-12 16:12:52,917 INFO  assembly.AggregateBy (AggregateBy.java:removeEldestEntry(327)) - flushed keys num times: 1, with threshold: 2
2015-02-12 16:12:52,917 INFO  assembly.AggregateBy (AggregateBy.java:removeEldestEntry(328)) - mem on flush (mb), free: 172, total: 513, max: 672
2015-02-12 16:12:52,917 INFO  assembly.AggregateBy (AggregateBy.java:removeEldestEntry(333)) - total mem is 76% of max mem, to better utilize unused memory consider increasing current LRU threshold with system property "cascading.aggregateby.threshold"
2015-02-12 16:12:52,918 INFO  assembly.AggregateBy (AggregateBy.java:removeEldestEntry(327)) - flushed keys num times: 3, with threshold: 2
2015-02-12 16:12:52,918 INFO  assembly.AggregateBy (AggregateBy.java:removeEldestEntry(328)) - mem on flush (mb), free: 172, total: 513, max: 672
2015-02-12 16:12:52,918 INFO  assembly.AggregateBy (AggregateBy.java:removeEldestEntry(333)) - total mem is 76% of max mem, to better utilize unused memory consider increasing current LRU threshold with system property "cascading.aggregateby.threshold"
2015-02-12 16:12:52,918 INFO  assembly.AggregateBy (AggregateBy.java:removeEldestEntry(327)) - flushed keys num times: 5, with threshold: 2
2015-02-12 16:12:52,918 INFO  assembly.AggregateBy (AggregateBy.java:removeEldestEntry(328)) - mem on flush (mb), free: 172, total: 513, max: 672
2015-02-12 16:12:52,918 INFO  assembly.AggregateBy (AggregateBy.java:removeEldestEntry(333)) - total mem is 76% of max mem, to better utilize unused memory consider increasing current LRU threshold with system property "cascading.aggregateby.threshold"
2015-02-12 16:12:52,919 INFO  assembly.AggregateBy (AggregateBy.java:removeEldestEntry(327)) - flushed keys num times: 7, with threshold: 2
2015-02-12 16:12:52,919 INFO  assembly.AggregateBy (AggregateBy.java:removeEldestEntry(328)) - mem on flush (mb), free: 172, total: 513, max: 672
2015-02-12 16:12:52,919 INFO  assembly.AggregateBy (AggregateBy.java:removeEldestEntry(333)) - total mem is 76% of max mem, to better utilize unused memory consider increasing current LRU threshold with system property "cascading.aggregateby.threshold"
2015-02-12 16:12:52,920 INFO  mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) - 
2015-02-12 16:12:52,920 INFO  mapred.MapTask (MapTask.java:flush(1457)) - Starting flush of map output
2015-02-12 16:12:52,920 INFO  mapred.MapTask (MapTask.java:flush(1475)) - Spilling map output
2015-02-12 16:12:52,920 INFO  mapred.MapTask (MapTask.java:flush(1476)) - bufstart = 0; bufend = 180; bufvoid = 104857600
2015-02-12 16:12:52,920 INFO  mapred.MapTask (MapTask.java:flush(1478)) - kvstart = 26214396(104857584); kvend = 26214364(104857456); length = 33/6553600
2015-02-12 16:12:52,922 INFO  mapred.MapTask (MapTask.java:sortAndSpill(1660)) - Finished spill 0
2015-02-12 16:12:52,923 INFO  mapred.Task (Task.java:done(1001)) - Task:attempt_local1666493537_0029_m_000000_0 is done. And is in the process of committing
2015-02-12 16:12:52,924 INFO  mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) - file:/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/classes/test/data/lhs.txt:0+52
2015-02-12 16:12:52,924 INFO  mapred.Task (Task.java:sendDone(1121)) - Task 'attempt_local1666493537_0029_m_000000_0' done.
2015-02-12 16:12:52,924 INFO  mapred.LocalJobRunner (LocalJobRunner.java:run(249)) - Finishing task: attempt_local1666493537_0029_m_000000_0
2015-02-12 16:12:52,924 INFO  mapred.LocalJobRunner (LocalJobRunner.java:runTasks(456)) - map task executor complete.
2015-02-12 16:12:52,925 INFO  mapred.LocalJobRunner (LocalJobRunner.java:runTasks(448)) - Waiting for reduce tasks
2015-02-12 16:12:52,925 INFO  mapred.LocalJobRunner (LocalJobRunner.java:run(302)) - Starting task: attempt_local1666493537_0029_r_000000_0
2015-02-12 16:12:52,926 INFO  util.ProcfsBasedProcessTree (ProcfsBasedProcessTree.java:isAvailable(181)) - ProcfsBasedProcessTree currently is supported only on Linux.
2015-02-12 16:12:52,926 INFO  mapred.Task (Task.java:initialize(587)) -  Using ResourceCalculatorProcessTree : null
2015-02-12 16:12:52,926 INFO  mapred.ReduceTask (ReduceTask.java:run(362)) - Using ShuffleConsumerPlugin: org.apache.hadoop.mapreduce.task.reduce.Shuffle@40542fa7
2015-02-12 16:12:52,926 INFO  reduce.MergeManagerImpl (MergeManagerImpl.java:<init>(195)) - MergerManager: memoryLimit=493250144, maxSingleShuffleLimit=123312536, mergeThreshold=325545120, ioSortFactor=10, memToMemMergeOutputsThreshold=10
2015-02-12 16:12:52,927 INFO  reduce.EventFetcher (EventFetcher.java:run(61)) - attempt_local1666493537_0029_r_000000_0 Thread started: EventFetcher for fetching Map Completion Events
2015-02-12 16:12:52,928 INFO  reduce.LocalFetcher (LocalFetcher.java:copyMapOutput(141)) - localfetcher#23 about to shuffle output of map attempt_local1666493537_0029_m_000000_0 decomp: 200 len: 204 to MEMORY
2015-02-12 16:12:52,928 INFO  reduce.InMemoryMapOutput (InMemoryMapOutput.java:shuffle(100)) - Read 200 bytes from map-output for attempt_local1666493537_0029_m_000000_0
2015-02-12 16:12:52,928 INFO  reduce.MergeManagerImpl (MergeManagerImpl.java:closeInMemoryFile(313)) - closeInMemoryFile -> map-output of size: 200, inMemoryMapOutputs.size() -> 1, commitMemory -> 0, usedMemory ->200
2015-02-12 16:12:52,929 INFO  reduce.EventFetcher (EventFetcher.java:run(76)) - EventFetcher is interrupted.. Returning
2015-02-12 16:12:52,929 INFO  mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) - 1 / 1 copied.
2015-02-12 16:12:52,929 INFO  reduce.MergeManagerImpl (MergeManagerImpl.java:finalMerge(673)) - finalMerge called with 1 in-memory map-outputs and 0 on-disk map-outputs
2015-02-12 16:12:52,930 INFO  mapred.Merger (Merger.java:merge(606)) - Merging 1 sorted segments
2015-02-12 16:12:52,931 INFO  mapred.Merger (Merger.java:merge(705)) - Down to the last merge-pass, with 1 segments left of total size: 191 bytes
2015-02-12 16:12:52,932 INFO  reduce.MergeManagerImpl (MergeManagerImpl.java:finalMerge(750)) - Merged 1 segments, 200 bytes to disk to satisfy reduce memory limit
2015-02-12 16:12:52,932 INFO  reduce.MergeManagerImpl (MergeManagerImpl.java:finalMerge(780)) - Merging 1 files, 204 bytes from disk
2015-02-12 16:12:52,933 INFO  reduce.MergeManagerImpl (MergeManagerImpl.java:finalMerge(795)) - Merging 0 segments, 0 bytes from memory into reduce
2015-02-12 16:12:52,933 INFO  mapred.Merger (Merger.java:merge(606)) - Merging 1 sorted segments
2015-02-12 16:12:52,933 INFO  mapred.Merger (Merger.java:merge(705)) - Down to the last merge-pass, with 1 segments left of total size: 191 bytes
2015-02-12 16:12:52,933 INFO  mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) - 1 / 1 copied.
2015-02-12 16:12:52,934 INFO  hadoop.FlowReducer (FlowReducer.java:configure(78)) - cascading version: 2.6.3
2015-02-12 16:12:52,934 INFO  hadoop.FlowReducer (FlowReducer.java:configure(79)) - child jvm opts: -Xmx200m
2015-02-12 16:12:52,937 INFO  hadoop.FlowReducer (FlowReducer.java:configure(97)) - sourcing from: GroupBy(average)[by:[{1}:'char']]
2015-02-12 16:12:52,937 INFO  hadoop.FlowReducer (FlowReducer.java:configure(100)) - sinking to: Hfs["TextDelimited[['char', 'average']]"]["/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/test/output/hadoop2-mr1/assemblyhelpersplatform/averagenull"]
2015-02-12 16:12:52,944 INFO  mapred.Task (Task.java:done(1001)) - Task:attempt_local1666493537_0029_r_000000_0 is done. And is in the process of committing
2015-02-12 16:12:52,945 INFO  mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) - 1 / 1 copied.
2015-02-12 16:12:52,945 INFO  mapred.Task (Task.java:commit(1162)) - Task attempt_local1666493537_0029_r_000000_0 is allowed to commit now
2015-02-12 16:12:52,946 INFO  output.FileOutputCommitter (FileOutputCommitter.java:commitTask(439)) - Saved output of task 'attempt_local1666493537_0029_r_000000_0' to file:/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/test/output/hadoop2-mr1/assemblyhelpersplatform/averagenull/_temporary/0/task_local1666493537_0029_r_000000
2015-02-12 16:12:52,946 INFO  mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) - reduce > reduce
2015-02-12 16:12:52,947 INFO  mapred.Task (Task.java:sendDone(1121)) - Task 'attempt_local1666493537_0029_r_000000_0' done.
2015-02-12 16:12:52,947 INFO  mapred.LocalJobRunner (LocalJobRunner.java:run(325)) - Finishing task: attempt_local1666493537_0029_r_000000_0
2015-02-12 16:12:52,947 INFO  mapred.LocalJobRunner (LocalJobRunner.java:runTasks(456)) - reduce task executor complete.
2015-02-12 16:12:52,957 INFO  util.Hadoop18TapUtil (Hadoop18TapUtil.java:cleanTempPath(219)) - deleting temp path /Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/test/output/hadoop2-mr1/assemblyhelpersplatform/averagenull/_temporary
2015-02-12 16:12:52,982 INFO  mapred.FileInputFormat (FileInputFormat.java:listStatus(247)) - Total input paths to process : 1
2015-02-12 16:12:53,005 INFO  mapred.FileInputFormat (FileInputFormat.java:listStatus(247)) - Total input paths to process : 1
2015-02-12 16:12:53,006 INFO  cascading.PlatformTestCase (PlatformTestCase.java:tearDown(209)) - copying to local /Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/test/output//hadoop2-mr1/assemblyhelpersplatform/averagenull
2015-02-12 16:12:53,017 INFO  util.HadoopUtil (HadoopUtil.java:findMainClass(336)) - using default application jar, may cause class not found exceptions on the cluster
2015-02-12 16:12:53,017 INFO  planner.HadoopPlanner (HadoopPlanner.java:initialize(225)) - using application jar: /Users/soberoi/.gradle/caches/modules-2/files-2.1/cascading/cascading-hadoop2-mr1/2.6.3/9ae016e2b8d246b9c1de9168e715d9f1b971aa4a/cascading-hadoop2-mr1-2.6.3.jar
2015-02-12 16:12:53,020 INFO  Configuration.deprecation (Configuration.java:warnOnceIfDeprecated(1022)) - mapred.used.genericoptionsparser is deprecated. Instead, use mapreduce.client.genericoptionsparser.used
2015-02-12 16:12:53,022 INFO  Configuration.deprecation (Configuration.java:warnOnceIfDeprecated(1022)) - mapred.job.tracker is deprecated. Instead, use mapreduce.jobtracker.address
2015-02-12 16:12:53,022 INFO  Configuration.deprecation (Configuration.java:warnOnceIfDeprecated(1022)) - mapred.output.compress is deprecated. Instead, use mapreduce.output.fileoutputformat.compress
2015-02-12 16:12:53,025 INFO  flow.Flow (BaseFlow.java:logInfo(1378)) - [coerce] starting
2015-02-12 16:12:53,026 INFO  flow.Flow (BaseFlow.java:logInfo(1378)) - [coerce]  source: Hfs["TextLine[['offset', 'line']->[ALL]]"]["/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/classes/test/data/lower.txt"]
2015-02-12 16:12:53,026 INFO  flow.Flow (BaseFlow.java:logInfo(1378)) - [coerce]  sink: Hfs["TextLine[['line']->['num', 'char']]"]["/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/test/output/hadoop2-mr1/assemblyhelpersplatform/coerce"]
2015-02-12 16:12:53,026 INFO  flow.Flow (BaseFlow.java:logInfo(1378)) - [coerce]  parallel execution is enabled: false
2015-02-12 16:12:53,026 INFO  flow.Flow (BaseFlow.java:logInfo(1378)) - [coerce]  starting jobs: 1
2015-02-12 16:12:53,026 INFO  flow.Flow (BaseFlow.java:logInfo(1378)) - [coerce]  allocating threads: 1
2015-02-12 16:12:53,026 INFO  flow.FlowStep (BaseFlowStep.java:logInfo(834)) - [coerce] starting step: (1/1) ...blyhelpersplatform/coerce
2015-02-12 16:12:53,027 INFO  jvm.JvmMetrics (JvmMetrics.java:init(71)) - Cannot initialize JVM Metrics with processName=JobTracker, sessionId= - already initialized
2015-02-12 16:12:53,028 INFO  jvm.JvmMetrics (JvmMetrics.java:init(71)) - Cannot initialize JVM Metrics with processName=JobTracker, sessionId= - already initialized
2015-02-12 16:12:53,047 INFO  mapred.FileInputFormat (FileInputFormat.java:listStatus(247)) - Total input paths to process : 1
2015-02-12 16:12:53,064 INFO  mapreduce.JobSubmitter (JobSubmitter.java:submitJobInternal(492)) - number of splits:1
2015-02-12 16:12:53,075 INFO  mapreduce.JobSubmitter (JobSubmitter.java:printTokens(575)) - Submitting tokens for job: job_local1609083427_0030
2015-02-12 16:12:53,084 WARN  conf.Configuration (Configuration.java:loadProperty(2510)) - file:/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/tmp/cascading/staging/soberoi1609083427/.staging/job_local1609083427_0030/job.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.retry.interval;  Ignoring.
2015-02-12 16:12:53,085 WARN  conf.Configuration (Configuration.java:loadProperty(2510)) - file:/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/tmp/cascading/staging/soberoi1609083427/.staging/job_local1609083427_0030/job.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.attempts;  Ignoring.
2015-02-12 16:12:53,118 WARN  conf.Configuration (Configuration.java:loadProperty(2510)) - file:/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/test/tmp/mapred/local/localRunner/soberoi/job_local1609083427_0030/job_local1609083427_0030.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.retry.interval;  Ignoring.
2015-02-12 16:12:53,120 WARN  conf.Configuration (Configuration.java:loadProperty(2510)) - file:/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/test/tmp/mapred/local/localRunner/soberoi/job_local1609083427_0030/job_local1609083427_0030.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.attempts;  Ignoring.
2015-02-12 16:12:53,120 INFO  mapreduce.Job (Job.java:submit(1299)) - The url to track the job: http://localhost:8080/
2015-02-12 16:12:53,121 INFO  flow.FlowStep (BaseFlowStep.java:logInfo(834)) - [coerce] submitted hadoop job: job_local1609083427_0030
2015-02-12 16:12:53,121 INFO  mapred.LocalJobRunner (LocalJobRunner.java:createOutputCommitter(471)) - OutputCommitter set in config null
2015-02-12 16:12:53,121 INFO  mapred.LocalJobRunner (LocalJobRunner.java:createOutputCommitter(489)) - OutputCommitter is org.apache.hadoop.mapred.FileOutputCommitter
2015-02-12 16:12:53,121 INFO  flow.FlowStep (BaseFlowStep.java:logInfo(834)) - [coerce] tracking url: http://localhost:8080/
2015-02-12 16:12:53,123 INFO  mapred.LocalJobRunner (LocalJobRunner.java:runTasks(448)) - Waiting for map tasks
2015-02-12 16:12:53,123 INFO  mapred.LocalJobRunner (LocalJobRunner.java:run(224)) - Starting task: attempt_local1609083427_0030_m_000000_0
2015-02-12 16:12:53,124 INFO  util.ProcfsBasedProcessTree (ProcfsBasedProcessTree.java:isAvailable(181)) - ProcfsBasedProcessTree currently is supported only on Linux.
2015-02-12 16:12:53,124 INFO  mapred.Task (Task.java:initialize(587)) -  Using ResourceCalculatorProcessTree : null
2015-02-12 16:12:53,125 INFO  io.MultiInputSplit (MultiInputSplit.java:readFields(161)) - current split input path: file:/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/classes/test/data/lower.txt
2015-02-12 16:12:53,125 INFO  mapred.MapTask (MapTask.java:updateJobWithSplit(482)) - Processing split: cascading.tap.hadoop.io.MultiInputSplit@4d15f20a
2015-02-12 16:12:53,126 INFO  mapred.MapTask (MapTask.java:runOldMapper(436)) - numReduceTasks: 0
2015-02-12 16:12:53,134 INFO  hadoop.FlowMapper (FlowMapper.java:configure(71)) - cascading version: 2.6.3
2015-02-12 16:12:53,134 INFO  hadoop.FlowMapper (FlowMapper.java:configure(72)) - child jvm opts: -Xmx200m
2015-02-12 16:12:53,136 INFO  hadoop.FlowMapper (FlowMapper.java:configure(87)) - sourcing from: Hfs["TextLine[['offset', 'line']->[ALL]]"]["/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/classes/test/data/lower.txt"]
2015-02-12 16:12:53,137 INFO  hadoop.FlowMapper (FlowMapper.java:configure(90)) - sinking to: Hfs["TextLine[['line']->['num', 'char']]"]["/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/test/output/hadoop2-mr1/assemblyhelpersplatform/coerce"]
2015-02-12 16:12:53,139 INFO  mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) - 
2015-02-12 16:12:53,139 INFO  mapred.Task (Task.java:done(1001)) - Task:attempt_local1609083427_0030_m_000000_0 is done. And is in the process of committing
2015-02-12 16:12:53,140 INFO  mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) - 
2015-02-12 16:12:53,140 INFO  mapred.Task (Task.java:commit(1162)) - Task attempt_local1609083427_0030_m_000000_0 is allowed to commit now
2015-02-12 16:12:53,141 INFO  output.FileOutputCommitter (FileOutputCommitter.java:commitTask(439)) - Saved output of task 'attempt_local1609083427_0030_m_000000_0' to file:/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/test/output/hadoop2-mr1/assemblyhelpersplatform/coerce/_temporary/0/task_local1609083427_0030_m_000000
2015-02-12 16:12:53,141 INFO  mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) - file:/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/classes/test/data/lower.txt:0+19
2015-02-12 16:12:53,142 INFO  mapred.Task (Task.java:sendDone(1121)) - Task 'attempt_local1609083427_0030_m_000000_0' done.
2015-02-12 16:12:53,142 INFO  mapred.LocalJobRunner (LocalJobRunner.java:run(249)) - Finishing task: attempt_local1609083427_0030_m_000000_0
2015-02-12 16:12:53,142 INFO  mapred.LocalJobRunner (LocalJobRunner.java:runTasks(456)) - map task executor complete.
2015-02-12 16:12:53,162 INFO  util.Hadoop18TapUtil (Hadoop18TapUtil.java:cleanTempPath(219)) - deleting temp path /Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/test/output/hadoop2-mr1/assemblyhelpersplatform/coerce/_temporary
2015-02-12 16:12:53,189 INFO  mapred.FileInputFormat (FileInputFormat.java:listStatus(247)) - Total input paths to process : 1
2015-02-12 16:12:53,190 INFO  cascading.PlatformTestCase (PlatformTestCase.java:tearDown(209)) - copying to local /Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/test/output//hadoop2-mr1/assemblyhelpersplatform/coerce
2015-02-12 16:12:53,208 INFO  util.HadoopUtil (HadoopUtil.java:findMainClass(336)) - using default application jar, may cause class not found exceptions on the cluster
2015-02-12 16:12:53,208 INFO  planner.HadoopPlanner (HadoopPlanner.java:initialize(225)) - using application jar: /Users/soberoi/.gradle/caches/modules-2/files-2.1/cascading/cascading-hadoop2-mr1/2.6.3/9ae016e2b8d246b9c1de9168e715d9f1b971aa4a/cascading-hadoop2-mr1-2.6.3.jar
2015-02-12 16:12:53,213 INFO  Configuration.deprecation (Configuration.java:warnOnceIfDeprecated(1022)) - mapred.used.genericoptionsparser is deprecated. Instead, use mapreduce.client.genericoptionsparser.used
2015-02-12 16:12:53,214 INFO  Configuration.deprecation (Configuration.java:warnOnceIfDeprecated(1022)) - mapred.job.tracker is deprecated. Instead, use mapreduce.jobtracker.address
2015-02-12 16:12:53,215 INFO  Configuration.deprecation (Configuration.java:warnOnceIfDeprecated(1022)) - mapred.output.compress is deprecated. Instead, use mapreduce.output.fileoutputformat.compress
2015-02-12 16:12:53,218 INFO  flow.Flow (BaseFlow.java:logInfo(1378)) - [first] starting
2015-02-12 16:12:53,218 INFO  flow.Flow (BaseFlow.java:logInfo(1378)) - [first]  source: Hfs["TextDelimited[['num', 'lower', 'upper']]"]["/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/classes/test/data/lhs+rhs-cross-rev.txt"]
2015-02-12 16:12:53,218 INFO  flow.Flow (BaseFlow.java:logInfo(1378)) - [first]  sink: Hfs["TextDelimited[['num', 'lower', 'upper']]"]["/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/test/output/hadoop2-mr1/assemblyhelpersplatform/firstnfieldswithoutcomparator"]
2015-02-12 16:12:53,218 INFO  flow.Flow (BaseFlow.java:logInfo(1378)) - [first]  parallel execution is enabled: false
2015-02-12 16:12:53,218 INFO  flow.Flow (BaseFlow.java:logInfo(1378)) - [first]  starting jobs: 1
2015-02-12 16:12:53,218 INFO  flow.Flow (BaseFlow.java:logInfo(1378)) - [first]  allocating threads: 1
2015-02-12 16:12:53,219 INFO  flow.FlowStep (BaseFlowStep.java:logInfo(834)) - [first] starting step: (1/1) ...tnfieldswithoutcomparator
2015-02-12 16:12:53,219 INFO  jvm.JvmMetrics (JvmMetrics.java:init(71)) - Cannot initialize JVM Metrics with processName=JobTracker, sessionId= - already initialized
2015-02-12 16:12:53,220 INFO  jvm.JvmMetrics (JvmMetrics.java:init(71)) - Cannot initialize JVM Metrics with processName=JobTracker, sessionId= - already initialized
2015-02-12 16:12:53,239 INFO  mapred.FileInputFormat (FileInputFormat.java:listStatus(247)) - Total input paths to process : 1
2015-02-12 16:12:53,255 INFO  mapreduce.JobSubmitter (JobSubmitter.java:submitJobInternal(492)) - number of splits:1
2015-02-12 16:12:53,267 INFO  mapreduce.JobSubmitter (JobSubmitter.java:printTokens(575)) - Submitting tokens for job: job_local45421486_0031
2015-02-12 16:12:53,277 WARN  conf.Configuration (Configuration.java:loadProperty(2510)) - file:/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/tmp/cascading/staging/soberoi45421486/.staging/job_local45421486_0031/job.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.retry.interval;  Ignoring.
2015-02-12 16:12:53,278 WARN  conf.Configuration (Configuration.java:loadProperty(2510)) - file:/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/tmp/cascading/staging/soberoi45421486/.staging/job_local45421486_0031/job.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.attempts;  Ignoring.
2015-02-12 16:12:53,307 WARN  conf.Configuration (Configuration.java:loadProperty(2510)) - file:/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/test/tmp/mapred/local/localRunner/soberoi/job_local45421486_0031/job_local45421486_0031.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.retry.interval;  Ignoring.
2015-02-12 16:12:53,308 WARN  conf.Configuration (Configuration.java:loadProperty(2510)) - file:/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/test/tmp/mapred/local/localRunner/soberoi/job_local45421486_0031/job_local45421486_0031.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.attempts;  Ignoring.
2015-02-12 16:12:53,309 INFO  mapreduce.Job (Job.java:submit(1299)) - The url to track the job: http://localhost:8080/
2015-02-12 16:12:53,314 INFO  mapred.LocalJobRunner (LocalJobRunner.java:createOutputCommitter(471)) - OutputCommitter set in config null
2015-02-12 16:12:53,314 INFO  mapred.LocalJobRunner (LocalJobRunner.java:createOutputCommitter(489)) - OutputCommitter is org.apache.hadoop.mapred.FileOutputCommitter
2015-02-12 16:12:53,314 INFO  flow.FlowStep (BaseFlowStep.java:logInfo(834)) - [first] submitted hadoop job: job_local45421486_0031
2015-02-12 16:12:53,315 INFO  flow.FlowStep (BaseFlowStep.java:logInfo(834)) - [first] tracking url: http://localhost:8080/
2015-02-12 16:12:53,316 INFO  mapred.LocalJobRunner (LocalJobRunner.java:runTasks(448)) - Waiting for map tasks
2015-02-12 16:12:53,316 INFO  mapred.LocalJobRunner (LocalJobRunner.java:run(224)) - Starting task: attempt_local45421486_0031_m_000000_0
2015-02-12 16:12:53,317 INFO  util.ProcfsBasedProcessTree (ProcfsBasedProcessTree.java:isAvailable(181)) - ProcfsBasedProcessTree currently is supported only on Linux.
2015-02-12 16:12:53,318 INFO  mapred.Task (Task.java:initialize(587)) -  Using ResourceCalculatorProcessTree : null
2015-02-12 16:12:53,319 INFO  io.MultiInputSplit (MultiInputSplit.java:readFields(161)) - current split input path: file:/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/classes/test/data/lhs+rhs-cross-rev.txt
2015-02-12 16:12:53,319 INFO  mapred.MapTask (MapTask.java:updateJobWithSplit(482)) - Processing split: cascading.tap.hadoop.io.MultiInputSplit@2a3df98e
2015-02-12 16:12:53,319 INFO  mapred.MapTask (MapTask.java:runOldMapper(436)) - numReduceTasks: 1
2015-02-12 16:12:53,331 INFO  mapred.MapTask (MapTask.java:setEquator(1202)) - (EQUATOR) 0 kvi 26214396(104857584)
2015-02-12 16:12:53,331 INFO  mapred.MapTask (MapTask.java:init(995)) - mapreduce.task.io.sort.mb: 100
2015-02-12 16:12:53,331 INFO  mapred.MapTask (MapTask.java:init(996)) - soft limit at 83886080
2015-02-12 16:12:53,331 INFO  mapred.MapTask (MapTask.java:init(997)) - bufstart = 0; bufvoid = 104857600
2015-02-12 16:12:53,331 INFO  mapred.MapTask (MapTask.java:init(998)) - kvstart = 26214396; length = 6553600
2015-02-12 16:12:53,332 INFO  mapred.MapTask (MapTask.java:createSortingCollector(402)) - Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer
2015-02-12 16:12:53,332 INFO  hadoop.FlowMapper (FlowMapper.java:configure(71)) - cascading version: 2.6.3
2015-02-12 16:12:53,332 INFO  hadoop.FlowMapper (FlowMapper.java:configure(72)) - child jvm opts: -Xmx200m
2015-02-12 16:12:53,335 INFO  hadoop.FlowMapper (FlowMapper.java:configure(87)) - sourcing from: Hfs["TextDelimited[['num', 'lower', 'upper']]"]["/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/classes/test/data/lhs+rhs-cross-rev.txt"]
2015-02-12 16:12:53,335 INFO  hadoop.FlowMapper (FlowMapper.java:configure(90)) - sinking to: GroupBy(first)[by:[{1}:'num']]
2015-02-12 16:12:53,336 INFO  assembly.AggregateBy (AggregateBy.java:prepare(267)) - using threshold value: 2
2015-02-12 16:12:53,337 INFO  assembly.AggregateBy (AggregateBy.java:removeEldestEntry(327)) - flushed keys num times: 1, with threshold: 2
2015-02-12 16:12:53,337 INFO  assembly.AggregateBy (AggregateBy.java:removeEldestEntry(328)) - mem on flush (mb), free: 193, total: 513, max: 672
2015-02-12 16:12:53,337 INFO  assembly.AggregateBy (AggregateBy.java:removeEldestEntry(333)) - total mem is 76% of max mem, to better utilize unused memory consider increasing current LRU threshold with system property "cascading.aggregateby.threshold"
2015-02-12 16:12:53,338 INFO  assembly.AggregateBy (AggregateBy.java:removeEldestEntry(327)) - flushed keys num times: 3, with threshold: 2
2015-02-12 16:12:53,338 INFO  assembly.AggregateBy (AggregateBy.java:removeEldestEntry(328)) - mem on flush (mb), free: 193, total: 513, max: 672
2015-02-12 16:12:53,338 INFO  assembly.AggregateBy (AggregateBy.java:removeEldestEntry(333)) - total mem is 76% of max mem, to better utilize unused memory consider increasing current LRU threshold with system property "cascading.aggregateby.threshold"
2015-02-12 16:12:53,339 INFO  mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) - 
2015-02-12 16:12:53,339 INFO  mapred.MapTask (MapTask.java:flush(1457)) - Starting flush of map output
2015-02-12 16:12:53,339 INFO  mapred.MapTask (MapTask.java:flush(1475)) - Spilling map output
2015-02-12 16:12:53,340 INFO  mapred.MapTask (MapTask.java:flush(1476)) - bufstart = 0; bufend = 105; bufvoid = 104857600
2015-02-12 16:12:53,340 INFO  mapred.MapTask (MapTask.java:flush(1478)) - kvstart = 26214396(104857584); kvend = 26214380(104857520); length = 17/6553600
2015-02-12 16:12:53,341 INFO  mapred.MapTask (MapTask.java:sortAndSpill(1660)) - Finished spill 0
2015-02-12 16:12:53,342 INFO  mapred.Task (Task.java:done(1001)) - Task:attempt_local45421486_0031_m_000000_0 is done. And is in the process of committing
2015-02-12 16:12:53,343 INFO  mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) - file:/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/classes/test/data/lhs+rhs-cross-rev.txt:0+221
2015-02-12 16:12:53,344 INFO  mapred.Task (Task.java:sendDone(1121)) - Task 'attempt_local45421486_0031_m_000000_0' done.
2015-02-12 16:12:53,344 INFO  mapred.LocalJobRunner (LocalJobRunner.java:run(249)) - Finishing task: attempt_local45421486_0031_m_000000_0
2015-02-12 16:12:53,344 INFO  mapred.LocalJobRunner (LocalJobRunner.java:runTasks(456)) - map task executor complete.
2015-02-12 16:12:53,344 INFO  mapred.LocalJobRunner (LocalJobRunner.java:runTasks(448)) - Waiting for reduce tasks
2015-02-12 16:12:53,344 INFO  mapred.LocalJobRunner (LocalJobRunner.java:run(302)) - Starting task: attempt_local45421486_0031_r_000000_0
2015-02-12 16:12:53,345 INFO  util.ProcfsBasedProcessTree (ProcfsBasedProcessTree.java:isAvailable(181)) - ProcfsBasedProcessTree currently is supported only on Linux.
2015-02-12 16:12:53,345 INFO  mapred.Task (Task.java:initialize(587)) -  Using ResourceCalculatorProcessTree : null
2015-02-12 16:12:53,345 INFO  mapred.ReduceTask (ReduceTask.java:run(362)) - Using ShuffleConsumerPlugin: org.apache.hadoop.mapreduce.task.reduce.Shuffle@1ee3e282
2015-02-12 16:12:53,346 INFO  reduce.MergeManagerImpl (MergeManagerImpl.java:<init>(195)) - MergerManager: memoryLimit=493250144, maxSingleShuffleLimit=123312536, mergeThreshold=325545120, ioSortFactor=10, memToMemMergeOutputsThreshold=10
2015-02-12 16:12:53,346 INFO  reduce.EventFetcher (EventFetcher.java:run(61)) - attempt_local45421486_0031_r_000000_0 Thread started: EventFetcher for fetching Map Completion Events
2015-02-12 16:12:53,347 INFO  reduce.LocalFetcher (LocalFetcher.java:copyMapOutput(141)) - localfetcher#24 about to shuffle output of map attempt_local45421486_0031_m_000000_0 decomp: 117 len: 121 to MEMORY
2015-02-12 16:12:53,347 INFO  reduce.InMemoryMapOutput (InMemoryMapOutput.java:shuffle(100)) - Read 117 bytes from map-output for attempt_local45421486_0031_m_000000_0
2015-02-12 16:12:53,347 INFO  reduce.MergeManagerImpl (MergeManagerImpl.java:closeInMemoryFile(313)) - closeInMemoryFile -> map-output of size: 117, inMemoryMapOutputs.size() -> 1, commitMemory -> 0, usedMemory ->117
2015-02-12 16:12:53,347 INFO  reduce.EventFetcher (EventFetcher.java:run(76)) - EventFetcher is interrupted.. Returning
2015-02-12 16:12:53,348 INFO  mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) - 1 / 1 copied.
2015-02-12 16:12:53,348 INFO  reduce.MergeManagerImpl (MergeManagerImpl.java:finalMerge(673)) - finalMerge called with 1 in-memory map-outputs and 0 on-disk map-outputs
2015-02-12 16:12:53,349 INFO  mapred.Merger (Merger.java:merge(606)) - Merging 1 sorted segments
2015-02-12 16:12:53,349 INFO  mapred.Merger (Merger.java:merge(705)) - Down to the last merge-pass, with 1 segments left of total size: 108 bytes
2015-02-12 16:12:53,350 INFO  reduce.MergeManagerImpl (MergeManagerImpl.java:finalMerge(750)) - Merged 1 segments, 117 bytes to disk to satisfy reduce memory limit
2015-02-12 16:12:53,350 INFO  reduce.MergeManagerImpl (MergeManagerImpl.java:finalMerge(780)) - Merging 1 files, 121 bytes from disk
2015-02-12 16:12:53,350 INFO  reduce.MergeManagerImpl (MergeManagerImpl.java:finalMerge(795)) - Merging 0 segments, 0 bytes from memory into reduce
2015-02-12 16:12:53,351 INFO  mapred.Merger (Merger.java:merge(606)) - Merging 1 sorted segments
2015-02-12 16:12:53,351 INFO  mapred.Merger (Merger.java:merge(705)) - Down to the last merge-pass, with 1 segments left of total size: 108 bytes
2015-02-12 16:12:53,351 INFO  mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) - 1 / 1 copied.
2015-02-12 16:12:53,352 INFO  hadoop.FlowReducer (FlowReducer.java:configure(78)) - cascading version: 2.6.3
2015-02-12 16:12:53,352 INFO  hadoop.FlowReducer (FlowReducer.java:configure(79)) - child jvm opts: -Xmx200m
2015-02-12 16:12:53,355 INFO  hadoop.FlowReducer (FlowReducer.java:configure(97)) - sourcing from: GroupBy(first)[by:[{1}:'num']]
2015-02-12 16:12:53,355 INFO  hadoop.FlowReducer (FlowReducer.java:configure(100)) - sinking to: Hfs["TextDelimited[['num', 'lower', 'upper']]"]["/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/test/output/hadoop2-mr1/assemblyhelpersplatform/firstnfieldswithoutcomparator"]
2015-02-12 16:12:53,361 INFO  mapred.Task (Task.java:done(1001)) - Task:attempt_local45421486_0031_r_000000_0 is done. And is in the process of committing
2015-02-12 16:12:53,362 INFO  mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) - 1 / 1 copied.
2015-02-12 16:12:53,362 INFO  mapred.Task (Task.java:commit(1162)) - Task attempt_local45421486_0031_r_000000_0 is allowed to commit now
2015-02-12 16:12:53,362 INFO  output.FileOutputCommitter (FileOutputCommitter.java:commitTask(439)) - Saved output of task 'attempt_local45421486_0031_r_000000_0' to file:/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/test/output/hadoop2-mr1/assemblyhelpersplatform/firstnfieldswithoutcomparator/_temporary/0/task_local45421486_0031_r_000000
2015-02-12 16:12:53,363 INFO  mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) - reduce > reduce
2015-02-12 16:12:53,363 INFO  mapred.Task (Task.java:sendDone(1121)) - Task 'attempt_local45421486_0031_r_000000_0' done.
2015-02-12 16:12:53,363 INFO  mapred.LocalJobRunner (LocalJobRunner.java:run(325)) - Finishing task: attempt_local45421486_0031_r_000000_0
2015-02-12 16:12:53,363 INFO  mapred.LocalJobRunner (LocalJobRunner.java:runTasks(456)) - reduce task executor complete.
2015-02-12 16:12:53,374 INFO  util.Hadoop18TapUtil (Hadoop18TapUtil.java:cleanTempPath(219)) - deleting temp path /Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/test/output/hadoop2-mr1/assemblyhelpersplatform/firstnfieldswithoutcomparator/_temporary
2015-02-12 16:12:53,394 INFO  mapred.FileInputFormat (FileInputFormat.java:listStatus(247)) - Total input paths to process : 1
2015-02-12 16:12:53,395 INFO  cascading.PlatformTestCase (PlatformTestCase.java:tearDown(209)) - copying to local /Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/test/output//hadoop2-mr1/assemblyhelpersplatform/firstnfieldswithoutcomparator
2015-02-12 16:12:53,407 INFO  util.HadoopUtil (HadoopUtil.java:findMainClass(336)) - using default application jar, may cause class not found exceptions on the cluster
2015-02-12 16:12:53,408 INFO  planner.HadoopPlanner (HadoopPlanner.java:initialize(225)) - using application jar: /Users/soberoi/.gradle/caches/modules-2/files-2.1/cascading/cascading-hadoop2-mr1/2.6.3/9ae016e2b8d246b9c1de9168e715d9f1b971aa4a/cascading-hadoop2-mr1-2.6.3.jar
2015-02-12 16:12:53,421 INFO  Configuration.deprecation (Configuration.java:warnOnceIfDeprecated(1022)) - mapred.used.genericoptionsparser is deprecated. Instead, use mapreduce.client.genericoptionsparser.used
2015-02-12 16:12:53,422 INFO  Configuration.deprecation (Configuration.java:warnOnceIfDeprecated(1022)) - mapred.job.tracker is deprecated. Instead, use mapreduce.jobtracker.address
2015-02-12 16:12:53,434 INFO  Configuration.deprecation (Configuration.java:warnOnceIfDeprecated(1022)) - mapred.output.compress is deprecated. Instead, use mapreduce.output.fileoutputformat.compress
2015-02-12 16:12:53,439 INFO  flow.Flow (BaseFlow.java:logInfo(1378)) - [count-lhs*source*count...] starting
2015-02-12 16:12:53,439 INFO  flow.Flow (BaseFlow.java:logInfo(1378)) - [count-lhs*source*count...]  source: Hfs["TextDelimited[['num', 'char']]"]["/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/classes/test/data/lhs.txt"]
2015-02-12 16:12:53,439 INFO  flow.Flow (BaseFlow.java:logInfo(1378)) - [count-lhs*source*count...]  sink: Hfs["TextDelimited[['char', 'count']]"]["/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/test/output/hadoop2-mr1/assemblyhelpersplatform/samesourcemergethreeway"]
2015-02-12 16:12:53,439 INFO  flow.Flow (BaseFlow.java:logInfo(1378)) - [count-lhs*source*count...]  parallel execution is enabled: false
2015-02-12 16:12:53,439 INFO  flow.Flow (BaseFlow.java:logInfo(1378)) - [count-lhs*source*count...]  starting jobs: 5
2015-02-12 16:12:53,439 INFO  flow.Flow (BaseFlow.java:logInfo(1378)) - [count-lhs*source*count...]  allocating threads: 1
2015-02-12 16:12:53,440 INFO  flow.FlowStep (BaseFlowStep.java:logInfo(834)) - [count-lhs*source*count...] starting step: (1/5)
2015-02-12 16:12:53,440 INFO  jvm.JvmMetrics (JvmMetrics.java:init(71)) - Cannot initialize JVM Metrics with processName=JobTracker, sessionId= - already initialized
2015-02-12 16:12:53,441 INFO  jvm.JvmMetrics (JvmMetrics.java:init(71)) - Cannot initialize JVM Metrics with processName=JobTracker, sessionId= - already initialized
2015-02-12 16:12:53,458 INFO  mapred.FileInputFormat (FileInputFormat.java:listStatus(247)) - Total input paths to process : 1
2015-02-12 16:12:53,472 INFO  mapreduce.JobSubmitter (JobSubmitter.java:submitJobInternal(492)) - number of splits:1
2015-02-12 16:12:53,484 INFO  mapreduce.JobSubmitter (JobSubmitter.java:printTokens(575)) - Submitting tokens for job: job_local2020268039_0032
2015-02-12 16:12:53,494 WARN  conf.Configuration (Configuration.java:loadProperty(2510)) - file:/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/tmp/cascading/staging/soberoi2020268039/.staging/job_local2020268039_0032/job.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.retry.interval;  Ignoring.
2015-02-12 16:12:53,495 WARN  conf.Configuration (Configuration.java:loadProperty(2510)) - file:/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/tmp/cascading/staging/soberoi2020268039/.staging/job_local2020268039_0032/job.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.attempts;  Ignoring.
2015-02-12 16:12:53,525 WARN  conf.Configuration (Configuration.java:loadProperty(2510)) - file:/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/test/tmp/mapred/local/localRunner/soberoi/job_local2020268039_0032/job_local2020268039_0032.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.retry.interval;  Ignoring.
2015-02-12 16:12:53,526 WARN  conf.Configuration (Configuration.java:loadProperty(2510)) - file:/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/test/tmp/mapred/local/localRunner/soberoi/job_local2020268039_0032/job_local2020268039_0032.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.attempts;  Ignoring.
2015-02-12 16:12:53,526 INFO  mapreduce.Job (Job.java:submit(1299)) - The url to track the job: http://localhost:8080/
2015-02-12 16:12:53,527 INFO  mapred.LocalJobRunner (LocalJobRunner.java:createOutputCommitter(471)) - OutputCommitter set in config null
2015-02-12 16:12:53,527 INFO  flow.FlowStep (BaseFlowStep.java:logInfo(834)) - [count-lhs*source*count...] submitted hadoop job: job_local2020268039_0032
2015-02-12 16:12:53,527 INFO  mapred.LocalJobRunner (LocalJobRunner.java:createOutputCommitter(489)) - OutputCommitter is org.apache.hadoop.mapred.FileOutputCommitter
2015-02-12 16:12:53,527 INFO  flow.FlowStep (BaseFlowStep.java:logInfo(834)) - [count-lhs*source*count...] tracking url: http://localhost:8080/
2015-02-12 16:12:53,529 INFO  mapred.LocalJobRunner (LocalJobRunner.java:runTasks(448)) - Waiting for map tasks
2015-02-12 16:12:53,529 INFO  mapred.LocalJobRunner (LocalJobRunner.java:run(224)) - Starting task: attempt_local2020268039_0032_m_000000_0
2015-02-12 16:12:53,530 INFO  util.ProcfsBasedProcessTree (ProcfsBasedProcessTree.java:isAvailable(181)) - ProcfsBasedProcessTree currently is supported only on Linux.
2015-02-12 16:12:53,530 INFO  mapred.Task (Task.java:initialize(587)) -  Using ResourceCalculatorProcessTree : null
2015-02-12 16:12:53,530 INFO  io.MultiInputSplit (MultiInputSplit.java:readFields(161)) - current split input path: file:/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/classes/test/data/lhs.txt
2015-02-12 16:12:53,530 INFO  mapred.MapTask (MapTask.java:updateJobWithSplit(482)) - Processing split: cascading.tap.hadoop.io.MultiInputSplit@6dfff19a
2015-02-12 16:12:53,531 INFO  mapred.MapTask (MapTask.java:runOldMapper(436)) - numReduceTasks: 1
2015-02-12 16:12:53,546 INFO  mapred.MapTask (MapTask.java:setEquator(1202)) - (EQUATOR) 0 kvi 26214396(104857584)
2015-02-12 16:12:53,546 INFO  mapred.MapTask (MapTask.java:init(995)) - mapreduce.task.io.sort.mb: 100
2015-02-12 16:12:53,547 INFO  mapred.MapTask (MapTask.java:init(996)) - soft limit at 83886080
2015-02-12 16:12:53,547 INFO  mapred.MapTask (MapTask.java:init(997)) - bufstart = 0; bufvoid = 104857600
2015-02-12 16:12:53,547 INFO  mapred.MapTask (MapTask.java:init(998)) - kvstart = 26214396; length = 6553600
2015-02-12 16:12:53,547 INFO  mapred.MapTask (MapTask.java:createSortingCollector(402)) - Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer
2015-02-12 16:12:53,548 INFO  hadoop.FlowMapper (FlowMapper.java:configure(71)) - cascading version: 2.6.3
2015-02-12 16:12:53,548 INFO  hadoop.FlowMapper (FlowMapper.java:configure(72)) - child jvm opts: -Xmx200m
2015-02-12 16:12:53,551 INFO  hadoop.FlowMapper (FlowMapper.java:configure(87)) - sourcing from: Hfs["TextDelimited[['num', 'char']]"]["/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/classes/test/data/lhs.txt"]
2015-02-12 16:12:53,551 INFO  hadoop.FlowMapper (FlowMapper.java:configure(90)) - sinking to: GroupBy(source)[by:[{1}:'char']]
2015-02-12 16:12:53,551 INFO  assembly.AggregateBy (AggregateBy.java:prepare(267)) - using threshold value: 2
2015-02-12 16:12:53,552 INFO  assembly.AggregateBy (AggregateBy.java:removeEldestEntry(327)) - flushed keys num times: 1, with threshold: 2
2015-02-12 16:12:53,552 INFO  assembly.AggregateBy (AggregateBy.java:removeEldestEntry(328)) - mem on flush (mb), free: 194, total: 513, max: 672
2015-02-12 16:12:53,552 INFO  assembly.AggregateBy (AggregateBy.java:removeEldestEntry(333)) - total mem is 76% of max mem, to better utilize unused memory consider increasing current LRU threshold with system property "cascading.aggregateby.threshold"
2015-02-12 16:12:53,552 INFO  assembly.AggregateBy (AggregateBy.java:removeEldestEntry(327)) - flushed keys num times: 3, with threshold: 2
2015-02-12 16:12:53,552 INFO  assembly.AggregateBy (AggregateBy.java:removeEldestEntry(328)) - mem on flush (mb), free: 194, total: 513, max: 672
2015-02-12 16:12:53,552 INFO  assembly.AggregateBy (AggregateBy.java:removeEldestEntry(333)) - total mem is 76% of max mem, to better utilize unused memory consider increasing current LRU threshold with system property "cascading.aggregateby.threshold"
2015-02-12 16:12:53,553 INFO  assembly.AggregateBy (AggregateBy.java:removeEldestEntry(327)) - flushed keys num times: 5, with threshold: 2
2015-02-12 16:12:53,553 INFO  assembly.AggregateBy (AggregateBy.java:removeEldestEntry(328)) - mem on flush (mb), free: 194, total: 513, max: 672
2015-02-12 16:12:53,553 INFO  assembly.AggregateBy (AggregateBy.java:removeEldestEntry(333)) - total mem is 76% of max mem, to better utilize unused memory consider increasing current LRU threshold with system property "cascading.aggregateby.threshold"
2015-02-12 16:12:53,553 INFO  assembly.AggregateBy (AggregateBy.java:removeEldestEntry(327)) - flushed keys num times: 7, with threshold: 2
2015-02-12 16:12:53,553 INFO  assembly.AggregateBy (AggregateBy.java:removeEldestEntry(328)) - mem on flush (mb), free: 193, total: 513, max: 672
2015-02-12 16:12:53,553 INFO  assembly.AggregateBy (AggregateBy.java:removeEldestEntry(333)) - total mem is 76% of max mem, to better utilize unused memory consider increasing current LRU threshold with system property "cascading.aggregateby.threshold"
2015-02-12 16:12:53,554 INFO  mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) - 
2015-02-12 16:12:53,554 INFO  mapred.MapTask (MapTask.java:flush(1457)) - Starting flush of map output
2015-02-12 16:12:53,554 INFO  mapred.MapTask (MapTask.java:flush(1475)) - Spilling map output
2015-02-12 16:12:53,554 INFO  mapred.MapTask (MapTask.java:flush(1476)) - bufstart = 0; bufend = 99; bufvoid = 104857600
2015-02-12 16:12:53,555 INFO  mapred.MapTask (MapTask.java:flush(1478)) - kvstart = 26214396(104857584); kvend = 26214364(104857456); length = 33/6553600
2015-02-12 16:12:53,556 INFO  mapred.MapTask (MapTask.java:sortAndSpill(1660)) - Finished spill 0
2015-02-12 16:12:53,557 INFO  mapred.Task (Task.java:done(1001)) - Task:attempt_local2020268039_0032_m_000000_0 is done. And is in the process of committing
2015-02-12 16:12:53,558 INFO  mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) - file:/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/classes/test/data/lhs.txt:0+52
2015-02-12 16:12:53,558 INFO  mapred.Task (Task.java:sendDone(1121)) - Task 'attempt_local2020268039_0032_m_000000_0' done.
2015-02-12 16:12:53,559 INFO  mapred.LocalJobRunner (LocalJobRunner.java:run(249)) - Finishing task: attempt_local2020268039_0032_m_000000_0
2015-02-12 16:12:53,559 INFO  mapred.LocalJobRunner (LocalJobRunner.java:runTasks(456)) - map task executor complete.
2015-02-12 16:12:53,559 INFO  mapred.LocalJobRunner (LocalJobRunner.java:runTasks(448)) - Waiting for reduce tasks
2015-02-12 16:12:53,559 INFO  mapred.LocalJobRunner (LocalJobRunner.java:run(302)) - Starting task: attempt_local2020268039_0032_r_000000_0
2015-02-12 16:12:53,560 INFO  util.ProcfsBasedProcessTree (ProcfsBasedProcessTree.java:isAvailable(181)) - ProcfsBasedProcessTree currently is supported only on Linux.
2015-02-12 16:12:53,560 INFO  mapred.Task (Task.java:initialize(587)) -  Using ResourceCalculatorProcessTree : null
2015-02-12 16:12:53,560 INFO  mapred.ReduceTask (ReduceTask.java:run(362)) - Using ShuffleConsumerPlugin: org.apache.hadoop.mapreduce.task.reduce.Shuffle@579ab801
2015-02-12 16:12:53,561 INFO  reduce.MergeManagerImpl (MergeManagerImpl.java:<init>(195)) - MergerManager: memoryLimit=493250144, maxSingleShuffleLimit=123312536, mergeThreshold=325545120, ioSortFactor=10, memToMemMergeOutputsThreshold=10
2015-02-12 16:12:53,561 INFO  reduce.EventFetcher (EventFetcher.java:run(61)) - attempt_local2020268039_0032_r_000000_0 Thread started: EventFetcher for fetching Map Completion Events
2015-02-12 16:12:53,562 INFO  reduce.LocalFetcher (LocalFetcher.java:copyMapOutput(141)) - localfetcher#25 about to shuffle output of map attempt_local2020268039_0032_m_000000_0 decomp: 119 len: 123 to MEMORY
2015-02-12 16:12:53,563 INFO  reduce.InMemoryMapOutput (InMemoryMapOutput.java:shuffle(100)) - Read 119 bytes from map-output for attempt_local2020268039_0032_m_000000_0
2015-02-12 16:12:53,563 INFO  reduce.MergeManagerImpl (MergeManagerImpl.java:closeInMemoryFile(313)) - closeInMemoryFile -> map-output of size: 119, inMemoryMapOutputs.size() -> 1, commitMemory -> 0, usedMemory ->119
2015-02-12 16:12:53,563 INFO  reduce.EventFetcher (EventFetcher.java:run(76)) - EventFetcher is interrupted.. Returning
2015-02-12 16:12:53,564 INFO  mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) - 1 / 1 copied.
2015-02-12 16:12:53,564 INFO  reduce.MergeManagerImpl (MergeManagerImpl.java:finalMerge(673)) - finalMerge called with 1 in-memory map-outputs and 0 on-disk map-outputs
2015-02-12 16:12:53,565 INFO  mapred.Merger (Merger.java:merge(606)) - Merging 1 sorted segments
2015-02-12 16:12:53,565 INFO  mapred.Merger (Merger.java:merge(705)) - Down to the last merge-pass, with 1 segments left of total size: 110 bytes
2015-02-12 16:12:53,566 INFO  reduce.MergeManagerImpl (MergeManagerImpl.java:finalMerge(750)) - Merged 1 segments, 119 bytes to disk to satisfy reduce memory limit
2015-02-12 16:12:53,566 INFO  reduce.MergeManagerImpl (MergeManagerImpl.java:finalMerge(780)) - Merging 1 files, 123 bytes from disk
2015-02-12 16:12:53,567 INFO  reduce.MergeManagerImpl (MergeManagerImpl.java:finalMerge(795)) - Merging 0 segments, 0 bytes from memory into reduce
2015-02-12 16:12:53,567 INFO  mapred.Merger (Merger.java:merge(606)) - Merging 1 sorted segments
2015-02-12 16:12:53,567 INFO  mapred.Merger (Merger.java:merge(705)) - Down to the last merge-pass, with 1 segments left of total size: 110 bytes
2015-02-12 16:12:53,567 INFO  mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) - 1 / 1 copied.
2015-02-12 16:12:53,568 INFO  hadoop.FlowReducer (FlowReducer.java:configure(78)) - cascading version: 2.6.3
2015-02-12 16:12:53,568 INFO  hadoop.FlowReducer (FlowReducer.java:configure(79)) - child jvm opts: -Xmx200m
2015-02-12 16:12:53,571 INFO  hadoop.FlowReducer (FlowReducer.java:configure(97)) - sourcing from: GroupBy(source)[by:[{1}:'char']]
2015-02-12 16:12:53,571 INFO  hadoop.FlowReducer (FlowReducer.java:configure(100)) - sinking to: TempHfs["SequenceFile[['char', 'count']]"][3798598062/source/]
2015-02-12 16:12:53,577 INFO  mapred.Task (Task.java:done(1001)) - Task:attempt_local2020268039_0032_r_000000_0 is done. And is in the process of committing
2015-02-12 16:12:53,578 INFO  mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) - 1 / 1 copied.
2015-02-12 16:12:53,578 INFO  mapred.Task (Task.java:commit(1162)) - Task attempt_local2020268039_0032_r_000000_0 is allowed to commit now
2015-02-12 16:12:53,579 INFO  output.FileOutputCommitter (FileOutputCommitter.java:commitTask(439)) - Saved output of task 'attempt_local2020268039_0032_r_000000_0' to file:/tmp/hadoop-soberoi/3798598062_source_BF20C2B770DC46CE8090B9EEF941B213/_temporary/0/task_local2020268039_0032_r_000000
2015-02-12 16:12:53,579 INFO  mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) - reduce > reduce
2015-02-12 16:12:53,580 INFO  mapred.Task (Task.java:sendDone(1121)) - Task 'attempt_local2020268039_0032_r_000000_0' done.
2015-02-12 16:12:53,580 INFO  mapred.LocalJobRunner (LocalJobRunner.java:run(325)) - Finishing task: attempt_local2020268039_0032_r_000000_0
2015-02-12 16:12:53,580 INFO  mapred.LocalJobRunner (LocalJobRunner.java:runTasks(456)) - reduce task executor complete.
2015-02-12 16:12:53,599 INFO  flow.FlowStep (BaseFlowStep.java:logInfo(834)) - [count-lhs*source*count...] starting step: (3/5)
2015-02-12 16:12:53,600 INFO  jvm.JvmMetrics (JvmMetrics.java:init(71)) - Cannot initialize JVM Metrics with processName=JobTracker, sessionId= - already initialized
2015-02-12 16:12:53,601 INFO  jvm.JvmMetrics (JvmMetrics.java:init(71)) - Cannot initialize JVM Metrics with processName=JobTracker, sessionId= - already initialized
2015-02-12 16:12:53,631 INFO  mapred.FileInputFormat (FileInputFormat.java:listStatus(247)) - Total input paths to process : 1
2015-02-12 16:12:53,646 INFO  mapreduce.JobSubmitter (JobSubmitter.java:submitJobInternal(492)) - number of splits:1
2015-02-12 16:12:53,655 INFO  Configuration.deprecation (Configuration.java:warnOnceIfDeprecated(1022)) - mapred.used.genericoptionsparser is deprecated. Instead, use mapreduce.client.genericoptionsparser.used
2015-02-12 16:12:53,655 INFO  Configuration.deprecation (Configuration.java:warnOnceIfDeprecated(1022)) - mapred.job.tracker is deprecated. Instead, use mapreduce.jobtracker.address
2015-02-12 16:12:53,658 INFO  mapreduce.JobSubmitter (JobSubmitter.java:printTokens(575)) - Submitting tokens for job: job_local1223674493_0033
2015-02-12 16:12:53,667 WARN  conf.Configuration (Configuration.java:loadProperty(2510)) - file:/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/tmp/cascading/staging/soberoi1223674493/.staging/job_local1223674493_0033/job.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.retry.interval;  Ignoring.
2015-02-12 16:12:53,668 WARN  conf.Configuration (Configuration.java:loadProperty(2510)) - file:/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/tmp/cascading/staging/soberoi1223674493/.staging/job_local1223674493_0033/job.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.attempts;  Ignoring.
2015-02-12 16:12:53,698 WARN  conf.Configuration (Configuration.java:loadProperty(2510)) - file:/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/test/tmp/mapred/local/localRunner/soberoi/job_local1223674493_0033/job_local1223674493_0033.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.retry.interval;  Ignoring.
2015-02-12 16:12:53,699 WARN  conf.Configuration (Configuration.java:loadProperty(2510)) - file:/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/test/tmp/mapred/local/localRunner/soberoi/job_local1223674493_0033/job_local1223674493_0033.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.attempts;  Ignoring.
2015-02-12 16:12:53,700 INFO  mapreduce.Job (Job.java:submit(1299)) - The url to track the job: http://localhost:8080/
2015-02-12 16:12:53,700 INFO  mapred.LocalJobRunner (LocalJobRunner.java:createOutputCommitter(471)) - OutputCommitter set in config null
2015-02-12 16:12:53,700 INFO  flow.FlowStep (BaseFlowStep.java:logInfo(834)) - [count-lhs*source*count...] submitted hadoop job: job_local1223674493_0033
2015-02-12 16:12:53,700 INFO  mapred.LocalJobRunner (LocalJobRunner.java:createOutputCommitter(489)) - OutputCommitter is org.apache.hadoop.mapred.FileOutputCommitter
2015-02-12 16:12:53,700 INFO  flow.FlowStep (BaseFlowStep.java:logInfo(834)) - [count-lhs*source*count...] tracking url: http://localhost:8080/
2015-02-12 16:12:53,701 INFO  mapred.LocalJobRunner (LocalJobRunner.java:runTasks(448)) - Waiting for map tasks
2015-02-12 16:12:53,702 INFO  mapred.LocalJobRunner (LocalJobRunner.java:run(224)) - Starting task: attempt_local1223674493_0033_m_000000_0
2015-02-12 16:12:53,702 INFO  util.ProcfsBasedProcessTree (ProcfsBasedProcessTree.java:isAvailable(181)) - ProcfsBasedProcessTree currently is supported only on Linux.
2015-02-12 16:12:53,703 INFO  mapred.Task (Task.java:initialize(587)) -  Using ResourceCalculatorProcessTree : null
2015-02-12 16:12:53,703 INFO  io.MultiInputSplit (MultiInputSplit.java:readFields(161)) - current split input path: file:/tmp/hadoop-soberoi/3798598062_source_BF20C2B770DC46CE8090B9EEF941B213/part-00000
2015-02-12 16:12:53,703 INFO  mapred.MapTask (MapTask.java:updateJobWithSplit(482)) - Processing split: cascading.tap.hadoop.io.MultiInputSplit@100a494b
2015-02-12 16:12:53,704 INFO  mapred.MapTask (MapTask.java:runOldMapper(436)) - numReduceTasks: 1
2015-02-12 16:12:53,720 INFO  mapred.MapTask (MapTask.java:setEquator(1202)) - (EQUATOR) 0 kvi 26214396(104857584)
2015-02-12 16:12:53,720 INFO  mapred.MapTask (MapTask.java:init(995)) - mapreduce.task.io.sort.mb: 100
2015-02-12 16:12:53,720 INFO  mapred.MapTask (MapTask.java:init(996)) - soft limit at 83886080
2015-02-12 16:12:53,720 INFO  mapred.MapTask (MapTask.java:init(997)) - bufstart = 0; bufvoid = 104857600
2015-02-12 16:12:53,720 INFO  mapred.MapTask (MapTask.java:init(998)) - kvstart = 26214396; length = 6553600
2015-02-12 16:12:53,721 INFO  mapred.MapTask (MapTask.java:createSortingCollector(402)) - Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer
2015-02-12 16:12:53,721 INFO  hadoop.FlowMapper (FlowMapper.java:configure(71)) - cascading version: 2.6.3
2015-02-12 16:12:53,722 INFO  hadoop.FlowMapper (FlowMapper.java:configure(72)) - child jvm opts: -Xmx200m
2015-02-12 16:12:53,725 INFO  hadoop.FlowMapper (FlowMapper.java:configure(87)) - sourcing from: TempHfs["SequenceFile[['char', 'count']]"][3798598062/source/]
2015-02-12 16:12:53,725 INFO  hadoop.FlowMapper (FlowMapper.java:configure(90)) - sinking to: GroupBy(count-lhs)[by:[{1}:'char']]
2015-02-12 16:12:53,725 INFO  assembly.AggregateBy (AggregateBy.java:prepare(267)) - using threshold value: 2
2015-02-12 16:12:53,726 INFO  assembly.AggregateBy (AggregateBy.java:removeEldestEntry(327)) - flushed keys num times: 1, with threshold: 2
2015-02-12 16:12:53,726 INFO  assembly.AggregateBy (AggregateBy.java:removeEldestEntry(328)) - mem on flush (mb), free: 195, total: 514, max: 672
2015-02-12 16:12:53,726 INFO  assembly.AggregateBy (AggregateBy.java:removeEldestEntry(333)) - total mem is 76% of max mem, to better utilize unused memory consider increasing current LRU threshold with system property "cascading.aggregateby.threshold"
2015-02-12 16:12:53,726 INFO  assembly.AggregateBy (AggregateBy.java:removeEldestEntry(327)) - flushed keys num times: 3, with threshold: 2
2015-02-12 16:12:53,726 INFO  assembly.AggregateBy (AggregateBy.java:removeEldestEntry(328)) - mem on flush (mb), free: 195, total: 514, max: 672
2015-02-12 16:12:53,727 INFO  assembly.AggregateBy (AggregateBy.java:removeEldestEntry(333)) - total mem is 76% of max mem, to better utilize unused memory consider increasing current LRU threshold with system property "cascading.aggregateby.threshold"
2015-02-12 16:12:53,727 INFO  mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) - 
2015-02-12 16:12:53,727 INFO  mapred.MapTask (MapTask.java:flush(1457)) - Starting flush of map output
2015-02-12 16:12:53,727 INFO  mapred.MapTask (MapTask.java:flush(1475)) - Spilling map output
2015-02-12 16:12:53,727 INFO  mapred.MapTask (MapTask.java:flush(1476)) - bufstart = 0; bufend = 55; bufvoid = 104857600
2015-02-12 16:12:53,728 INFO  mapred.MapTask (MapTask.java:flush(1478)) - kvstart = 26214396(104857584); kvend = 26214380(104857520); length = 17/6553600
2015-02-12 16:12:53,729 INFO  mapred.MapTask (MapTask.java:sortAndSpill(1660)) - Finished spill 0
2015-02-12 16:12:53,730 INFO  mapred.Task (Task.java:done(1001)) - Task:attempt_local1223674493_0033_m_000000_0 is done. And is in the process of committing
2015-02-12 16:12:53,731 INFO  mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) - file:/tmp/hadoop-soberoi/3798598062_source_BF20C2B770DC46CE8090B9EEF941B213/part-00000:0+160
2015-02-12 16:12:53,731 INFO  mapred.Task (Task.java:sendDone(1121)) - Task 'attempt_local1223674493_0033_m_000000_0' done.
2015-02-12 16:12:53,731 INFO  mapred.LocalJobRunner (LocalJobRunner.java:run(249)) - Finishing task: attempt_local1223674493_0033_m_000000_0
2015-02-12 16:12:53,731 INFO  mapred.LocalJobRunner (LocalJobRunner.java:runTasks(456)) - map task executor complete.
2015-02-12 16:12:53,732 INFO  mapred.LocalJobRunner (LocalJobRunner.java:runTasks(448)) - Waiting for reduce tasks
2015-02-12 16:12:53,732 INFO  mapred.LocalJobRunner (LocalJobRunner.java:run(302)) - Starting task: attempt_local1223674493_0033_r_000000_0
2015-02-12 16:12:53,733 INFO  util.ProcfsBasedProcessTree (ProcfsBasedProcessTree.java:isAvailable(181)) - ProcfsBasedProcessTree currently is supported only on Linux.
2015-02-12 16:12:53,733 INFO  mapred.Task (Task.java:initialize(587)) -  Using ResourceCalculatorProcessTree : null
2015-02-12 16:12:53,733 INFO  mapred.ReduceTask (ReduceTask.java:run(362)) - Using ShuffleConsumerPlugin: org.apache.hadoop.mapreduce.task.reduce.Shuffle@32f2e8ca
2015-02-12 16:12:53,733 INFO  reduce.MergeManagerImpl (MergeManagerImpl.java:<init>(195)) - MergerManager: memoryLimit=493250144, maxSingleShuffleLimit=123312536, mergeThreshold=325545120, ioSortFactor=10, memToMemMergeOutputsThreshold=10
2015-02-12 16:12:53,734 INFO  reduce.EventFetcher (EventFetcher.java:run(61)) - attempt_local1223674493_0033_r_000000_0 Thread started: EventFetcher for fetching Map Completion Events
2015-02-12 16:12:53,735 INFO  reduce.LocalFetcher (LocalFetcher.java:copyMapOutput(141)) - localfetcher#26 about to shuffle output of map attempt_local1223674493_0033_m_000000_0 decomp: 67 len: 71 to MEMORY
2015-02-12 16:12:53,735 INFO  reduce.InMemoryMapOutput (InMemoryMapOutput.java:shuffle(100)) - Read 67 bytes from map-output for attempt_local1223674493_0033_m_000000_0
2015-02-12 16:12:53,735 INFO  reduce.MergeManagerImpl (MergeManagerImpl.java:closeInMemoryFile(313)) - closeInMemoryFile -> map-output of size: 67, inMemoryMapOutputs.size() -> 1, commitMemory -> 0, usedMemory ->67
2015-02-12 16:12:53,736 INFO  reduce.EventFetcher (EventFetcher.java:run(76)) - EventFetcher is interrupted.. Returning
2015-02-12 16:12:53,736 INFO  mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) - 1 / 1 copied.
2015-02-12 16:12:53,736 INFO  reduce.MergeManagerImpl (MergeManagerImpl.java:finalMerge(673)) - finalMerge called with 1 in-memory map-outputs and 0 on-disk map-outputs
2015-02-12 16:12:53,737 INFO  mapred.Merger (Merger.java:merge(606)) - Merging 1 sorted segments
2015-02-12 16:12:53,738 INFO  mapred.Merger (Merger.java:merge(705)) - Down to the last merge-pass, with 1 segments left of total size: 58 bytes
2015-02-12 16:12:53,739 INFO  reduce.MergeManagerImpl (MergeManagerImpl.java:finalMerge(750)) - Merged 1 segments, 67 bytes to disk to satisfy reduce memory limit
2015-02-12 16:12:53,739 INFO  reduce.MergeManagerImpl (MergeManagerImpl.java:finalMerge(780)) - Merging 1 files, 71 bytes from disk
2015-02-12 16:12:53,739 INFO  reduce.MergeManagerImpl (MergeManagerImpl.java:finalMerge(795)) - Merging 0 segments, 0 bytes from memory into reduce
2015-02-12 16:12:53,739 INFO  mapred.Merger (Merger.java:merge(606)) - Merging 1 sorted segments
2015-02-12 16:12:53,740 INFO  mapred.Merger (Merger.java:merge(705)) - Down to the last merge-pass, with 1 segments left of total size: 58 bytes
2015-02-12 16:12:53,740 INFO  mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) - 1 / 1 copied.
2015-02-12 16:12:53,740 INFO  hadoop.FlowReducer (FlowReducer.java:configure(78)) - cascading version: 2.6.3
2015-02-12 16:12:53,740 INFO  hadoop.FlowReducer (FlowReducer.java:configure(79)) - child jvm opts: -Xmx200m
2015-02-12 16:12:53,744 INFO  hadoop.FlowReducer (FlowReducer.java:configure(97)) - sourcing from: GroupBy(count-lhs)[by:[{1}:'char']]
2015-02-12 16:12:53,744 INFO  hadoop.FlowReducer (FlowReducer.java:configure(100)) - sinking to: TempHfs["SequenceFile[['char', 'count']]"][6668818197/count-lhs/]
2015-02-12 16:12:53,751 INFO  mapred.Task (Task.java:done(1001)) - Task:attempt_local1223674493_0033_r_000000_0 is done. And is in the process of committing
2015-02-12 16:12:53,752 INFO  mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) - 1 / 1 copied.
2015-02-12 16:12:53,753 INFO  mapred.Task (Task.java:commit(1162)) - Task attempt_local1223674493_0033_r_000000_0 is allowed to commit now
2015-02-12 16:12:53,753 INFO  output.FileOutputCommitter (FileOutputCommitter.java:commitTask(439)) - Saved output of task 'attempt_local1223674493_0033_r_000000_0' to file:/tmp/hadoop-soberoi/6668818197_count_lhs_E0299A91C9AD4B61A1E657A2086B2F4E/_temporary/0/task_local1223674493_0033_r_000000
2015-02-12 16:12:53,754 INFO  mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) - reduce > reduce
2015-02-12 16:12:53,754 INFO  mapred.Task (Task.java:sendDone(1121)) - Task 'attempt_local1223674493_0033_r_000000_0' done.
2015-02-12 16:12:53,754 INFO  mapred.LocalJobRunner (LocalJobRunner.java:run(325)) - Finishing task: attempt_local1223674493_0033_r_000000_0
2015-02-12 16:12:53,755 INFO  mapred.LocalJobRunner (LocalJobRunner.java:runTasks(456)) - reduce task executor complete.
2015-02-12 16:12:53,773 INFO  flow.FlowStep (BaseFlowStep.java:logInfo(834)) - [count-lhs*source*count...] starting step: (4/5)
2015-02-12 16:12:53,773 INFO  jvm.JvmMetrics (JvmMetrics.java:init(71)) - Cannot initialize JVM Metrics with processName=JobTracker, sessionId= - already initialized
2015-02-12 16:12:53,774 INFO  jvm.JvmMetrics (JvmMetrics.java:init(71)) - Cannot initialize JVM Metrics with processName=JobTracker, sessionId= - already initialized
2015-02-12 16:12:53,803 INFO  mapred.FileInputFormat (FileInputFormat.java:listStatus(247)) - Total input paths to process : 1
2015-02-12 16:12:53,818 INFO  mapreduce.JobSubmitter (JobSubmitter.java:submitJobInternal(492)) - number of splits:1
2015-02-12 16:12:53,827 INFO  Configuration.deprecation (Configuration.java:warnOnceIfDeprecated(1022)) - mapred.used.genericoptionsparser is deprecated. Instead, use mapreduce.client.genericoptionsparser.used
2015-02-12 16:12:53,827 INFO  Configuration.deprecation (Configuration.java:warnOnceIfDeprecated(1022)) - mapred.job.tracker is deprecated. Instead, use mapreduce.jobtracker.address
2015-02-12 16:12:53,831 INFO  mapreduce.JobSubmitter (JobSubmitter.java:printTokens(575)) - Submitting tokens for job: job_local502182601_0034
2015-02-12 16:12:53,841 WARN  conf.Configuration (Configuration.java:loadProperty(2510)) - file:/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/tmp/cascading/staging/soberoi502182601/.staging/job_local502182601_0034/job.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.retry.interval;  Ignoring.
2015-02-12 16:12:53,842 WARN  conf.Configuration (Configuration.java:loadProperty(2510)) - file:/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/tmp/cascading/staging/soberoi502182601/.staging/job_local502182601_0034/job.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.attempts;  Ignoring.
2015-02-12 16:12:53,870 WARN  conf.Configuration (Configuration.java:loadProperty(2510)) - file:/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/test/tmp/mapred/local/localRunner/soberoi/job_local502182601_0034/job_local502182601_0034.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.retry.interval;  Ignoring.
2015-02-12 16:12:53,871 WARN  conf.Configuration (Configuration.java:loadProperty(2510)) - file:/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/test/tmp/mapred/local/localRunner/soberoi/job_local502182601_0034/job_local502182601_0034.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.attempts;  Ignoring.
2015-02-12 16:12:53,872 INFO  mapreduce.Job (Job.java:submit(1299)) - The url to track the job: http://localhost:8080/
2015-02-12 16:12:53,872 INFO  mapred.LocalJobRunner (LocalJobRunner.java:createOutputCommitter(471)) - OutputCommitter set in config null
2015-02-12 16:12:53,872 INFO  flow.FlowStep (BaseFlowStep.java:logInfo(834)) - [count-lhs*source*count...] submitted hadoop job: job_local502182601_0034
2015-02-12 16:12:53,872 INFO  mapred.LocalJobRunner (LocalJobRunner.java:createOutputCommitter(489)) - OutputCommitter is org.apache.hadoop.mapred.FileOutputCommitter
2015-02-12 16:12:53,872 INFO  flow.FlowStep (BaseFlowStep.java:logInfo(834)) - [count-lhs*source*count...] tracking url: http://localhost:8080/
2015-02-12 16:12:53,874 INFO  mapred.LocalJobRunner (LocalJobRunner.java:runTasks(448)) - Waiting for map tasks
2015-02-12 16:12:53,874 INFO  mapred.LocalJobRunner (LocalJobRunner.java:run(224)) - Starting task: attempt_local502182601_0034_m_000000_0
2015-02-12 16:12:53,875 INFO  util.ProcfsBasedProcessTree (ProcfsBasedProcessTree.java:isAvailable(181)) - ProcfsBasedProcessTree currently is supported only on Linux.
2015-02-12 16:12:53,875 INFO  mapred.Task (Task.java:initialize(587)) -  Using ResourceCalculatorProcessTree : null
2015-02-12 16:12:53,875 INFO  io.MultiInputSplit (MultiInputSplit.java:readFields(161)) - current split input path: file:/tmp/hadoop-soberoi/3798598062_source_BF20C2B770DC46CE8090B9EEF941B213/part-00000
2015-02-12 16:12:53,876 INFO  mapred.MapTask (MapTask.java:updateJobWithSplit(482)) - Processing split: cascading.tap.hadoop.io.MultiInputSplit@1854dfcf
2015-02-12 16:12:53,876 INFO  mapred.MapTask (MapTask.java:runOldMapper(436)) - numReduceTasks: 1
2015-02-12 16:12:53,892 INFO  mapred.MapTask (MapTask.java:setEquator(1202)) - (EQUATOR) 0 kvi 26214396(104857584)
2015-02-12 16:12:53,892 INFO  mapred.MapTask (MapTask.java:init(995)) - mapreduce.task.io.sort.mb: 100
2015-02-12 16:12:53,892 INFO  mapred.MapTask (MapTask.java:init(996)) - soft limit at 83886080
2015-02-12 16:12:53,892 INFO  mapred.MapTask (MapTask.java:init(997)) - bufstart = 0; bufvoid = 104857600
2015-02-12 16:12:53,892 INFO  mapred.MapTask (MapTask.java:init(998)) - kvstart = 26214396; length = 6553600
2015-02-12 16:12:53,893 INFO  mapred.MapTask (MapTask.java:createSortingCollector(402)) - Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer
2015-02-12 16:12:53,893 INFO  hadoop.FlowMapper (FlowMapper.java:configure(71)) - cascading version: 2.6.3
2015-02-12 16:12:53,893 INFO  hadoop.FlowMapper (FlowMapper.java:configure(72)) - child jvm opts: -Xmx200m
2015-02-12 16:12:53,896 INFO  hadoop.FlowMapper (FlowMapper.java:configure(87)) - sourcing from: TempHfs["SequenceFile[['char', 'count']]"][3798598062/source/]
2015-02-12 16:12:53,896 INFO  hadoop.FlowMapper (FlowMapper.java:configure(90)) - sinking to: GroupBy(count-rhs)[by:[{1}:'char']]
2015-02-12 16:12:53,897 INFO  assembly.AggregateBy (AggregateBy.java:prepare(267)) - using threshold value: 2
2015-02-12 16:12:53,898 INFO  assembly.AggregateBy (AggregateBy.java:removeEldestEntry(327)) - flushed keys num times: 1, with threshold: 2
2015-02-12 16:12:53,898 INFO  assembly.AggregateBy (AggregateBy.java:removeEldestEntry(328)) - mem on flush (mb), free: 195, total: 514, max: 672
2015-02-12 16:12:53,898 INFO  assembly.AggregateBy (AggregateBy.java:removeEldestEntry(333)) - total mem is 76% of max mem, to better utilize unused memory consider increasing current LRU threshold with system property "cascading.aggregateby.threshold"
2015-02-12 16:12:53,898 INFO  assembly.AggregateBy (AggregateBy.java:removeEldestEntry(327)) - flushed keys num times: 3, with threshold: 2
2015-02-12 16:12:53,898 INFO  assembly.AggregateBy (AggregateBy.java:removeEldestEntry(328)) - mem on flush (mb), free: 195, total: 514, max: 672
2015-02-12 16:12:53,898 INFO  assembly.AggregateBy (AggregateBy.java:removeEldestEntry(333)) - total mem is 76% of max mem, to better utilize unused memory consider increasing current LRU threshold with system property "cascading.aggregateby.threshold"
2015-02-12 16:12:53,899 INFO  mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) - 
2015-02-12 16:12:53,899 INFO  mapred.MapTask (MapTask.java:flush(1457)) - Starting flush of map output
2015-02-12 16:12:53,899 INFO  mapred.MapTask (MapTask.java:flush(1475)) - Spilling map output
2015-02-12 16:12:53,899 INFO  mapred.MapTask (MapTask.java:flush(1476)) - bufstart = 0; bufend = 55; bufvoid = 104857600
2015-02-12 16:12:53,899 INFO  mapred.MapTask (MapTask.java:flush(1478)) - kvstart = 26214396(104857584); kvend = 26214380(104857520); length = 17/6553600
2015-02-12 16:12:53,901 INFO  mapred.MapTask (MapTask.java:sortAndSpill(1660)) - Finished spill 0
2015-02-12 16:12:53,902 INFO  mapred.Task (Task.java:done(1001)) - Task:attempt_local502182601_0034_m_000000_0 is done. And is in the process of committing
2015-02-12 16:12:53,903 INFO  mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) - file:/tmp/hadoop-soberoi/3798598062_source_BF20C2B770DC46CE8090B9EEF941B213/part-00000:0+160
2015-02-12 16:12:53,903 INFO  mapred.Task (Task.java:sendDone(1121)) - Task 'attempt_local502182601_0034_m_000000_0' done.
2015-02-12 16:12:53,903 INFO  mapred.LocalJobRunner (LocalJobRunner.java:run(249)) - Finishing task: attempt_local502182601_0034_m_000000_0
2015-02-12 16:12:53,903 INFO  mapred.LocalJobRunner (LocalJobRunner.java:runTasks(456)) - map task executor complete.
2015-02-12 16:12:53,904 INFO  mapred.LocalJobRunner (LocalJobRunner.java:runTasks(448)) - Waiting for reduce tasks
2015-02-12 16:12:53,904 INFO  mapred.LocalJobRunner (LocalJobRunner.java:run(302)) - Starting task: attempt_local502182601_0034_r_000000_0
2015-02-12 16:12:53,905 INFO  util.ProcfsBasedProcessTree (ProcfsBasedProcessTree.java:isAvailable(181)) - ProcfsBasedProcessTree currently is supported only on Linux.
2015-02-12 16:12:53,905 INFO  mapred.Task (Task.java:initialize(587)) -  Using ResourceCalculatorProcessTree : null
2015-02-12 16:12:53,905 INFO  mapred.ReduceTask (ReduceTask.java:run(362)) - Using ShuffleConsumerPlugin: org.apache.hadoop.mapreduce.task.reduce.Shuffle@784c0b8
2015-02-12 16:12:53,905 INFO  reduce.MergeManagerImpl (MergeManagerImpl.java:<init>(195)) - MergerManager: memoryLimit=493250144, maxSingleShuffleLimit=123312536, mergeThreshold=325545120, ioSortFactor=10, memToMemMergeOutputsThreshold=10
2015-02-12 16:12:53,906 INFO  reduce.EventFetcher (EventFetcher.java:run(61)) - attempt_local502182601_0034_r_000000_0 Thread started: EventFetcher for fetching Map Completion Events
2015-02-12 16:12:53,906 INFO  reduce.LocalFetcher (LocalFetcher.java:copyMapOutput(141)) - localfetcher#27 about to shuffle output of map attempt_local502182601_0034_m_000000_0 decomp: 67 len: 71 to MEMORY
2015-02-12 16:12:53,907 INFO  reduce.InMemoryMapOutput (InMemoryMapOutput.java:shuffle(100)) - Read 67 bytes from map-output for attempt_local502182601_0034_m_000000_0
2015-02-12 16:12:53,907 INFO  reduce.MergeManagerImpl (MergeManagerImpl.java:closeInMemoryFile(313)) - closeInMemoryFile -> map-output of size: 67, inMemoryMapOutputs.size() -> 1, commitMemory -> 0, usedMemory ->67
2015-02-12 16:12:53,907 INFO  reduce.EventFetcher (EventFetcher.java:run(76)) - EventFetcher is interrupted.. Returning
2015-02-12 16:12:53,908 INFO  mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) - 1 / 1 copied.
2015-02-12 16:12:53,908 INFO  reduce.MergeManagerImpl (MergeManagerImpl.java:finalMerge(673)) - finalMerge called with 1 in-memory map-outputs and 0 on-disk map-outputs
2015-02-12 16:12:53,909 INFO  mapred.Merger (Merger.java:merge(606)) - Merging 1 sorted segments
2015-02-12 16:12:53,909 INFO  mapred.Merger (Merger.java:merge(705)) - Down to the last merge-pass, with 1 segments left of total size: 58 bytes
2015-02-12 16:12:53,910 INFO  reduce.MergeManagerImpl (MergeManagerImpl.java:finalMerge(750)) - Merged 1 segments, 67 bytes to disk to satisfy reduce memory limit
2015-02-12 16:12:53,911 INFO  reduce.MergeManagerImpl (MergeManagerImpl.java:finalMerge(780)) - Merging 1 files, 71 bytes from disk
2015-02-12 16:12:53,911 INFO  reduce.MergeManagerImpl (MergeManagerImpl.java:finalMerge(795)) - Merging 0 segments, 0 bytes from memory into reduce
2015-02-12 16:12:53,911 INFO  mapred.Merger (Merger.java:merge(606)) - Merging 1 sorted segments
2015-02-12 16:12:53,911 INFO  mapred.Merger (Merger.java:merge(705)) - Down to the last merge-pass, with 1 segments left of total size: 58 bytes
2015-02-12 16:12:53,912 INFO  mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) - 1 / 1 copied.
2015-02-12 16:12:53,912 INFO  hadoop.FlowReducer (FlowReducer.java:configure(78)) - cascading version: 2.6.3
2015-02-12 16:12:53,912 INFO  hadoop.FlowReducer (FlowReducer.java:configure(79)) - child jvm opts: -Xmx200m
2015-02-12 16:12:53,915 INFO  hadoop.FlowReducer (FlowReducer.java:configure(97)) - sourcing from: GroupBy(count-rhs)[by:[{1}:'char']]
2015-02-12 16:12:53,915 INFO  hadoop.FlowReducer (FlowReducer.java:configure(100)) - sinking to: TempHfs["SequenceFile[['char', 'count']]"][3761761564/count-rhs/]
2015-02-12 16:12:53,921 INFO  mapred.Task (Task.java:done(1001)) - Task:attempt_local502182601_0034_r_000000_0 is done. And is in the process of committing
2015-02-12 16:12:53,922 INFO  mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) - 1 / 1 copied.
2015-02-12 16:12:53,923 INFO  mapred.Task (Task.java:commit(1162)) - Task attempt_local502182601_0034_r_000000_0 is allowed to commit now
2015-02-12 16:12:53,923 INFO  output.FileOutputCommitter (FileOutputCommitter.java:commitTask(439)) - Saved output of task 'attempt_local502182601_0034_r_000000_0' to file:/tmp/hadoop-soberoi/3761761564_count_rhs_F5332AAA9616477EAC6E92C77907B8AC/_temporary/0/task_local502182601_0034_r_000000
2015-02-12 16:12:53,924 INFO  mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) - reduce > reduce
2015-02-12 16:12:53,924 INFO  mapred.Task (Task.java:sendDone(1121)) - Task 'attempt_local502182601_0034_r_000000_0' done.
2015-02-12 16:12:53,924 INFO  mapred.LocalJobRunner (LocalJobRunner.java:run(325)) - Finishing task: attempt_local502182601_0034_r_000000_0
2015-02-12 16:12:53,924 INFO  mapred.LocalJobRunner (LocalJobRunner.java:runTasks(456)) - reduce task executor complete.
2015-02-12 16:12:53,933 INFO  flow.FlowStep (BaseFlowStep.java:logInfo(834)) - [count-lhs*source*count...] starting step: (2/5)
2015-02-12 16:12:53,933 INFO  jvm.JvmMetrics (JvmMetrics.java:init(71)) - Cannot initialize JVM Metrics with processName=JobTracker, sessionId= - already initialized
2015-02-12 16:12:53,934 INFO  jvm.JvmMetrics (JvmMetrics.java:init(71)) - Cannot initialize JVM Metrics with processName=JobTracker, sessionId= - already initialized
2015-02-12 16:12:53,966 INFO  mapred.FileInputFormat (FileInputFormat.java:listStatus(247)) - Total input paths to process : 1
2015-02-12 16:12:53,977 INFO  mapred.FileInputFormat (FileInputFormat.java:listStatus(247)) - Total input paths to process : 1
2015-02-12 16:12:53,993 INFO  mapreduce.JobSubmitter (JobSubmitter.java:submitJobInternal(492)) - number of splits:2
2015-02-12 16:12:54,001 INFO  Configuration.deprecation (Configuration.java:warnOnceIfDeprecated(1022)) - mapred.used.genericoptionsparser is deprecated. Instead, use mapreduce.client.genericoptionsparser.used
2015-02-12 16:12:54,002 INFO  Configuration.deprecation (Configuration.java:warnOnceIfDeprecated(1022)) - mapred.job.tracker is deprecated. Instead, use mapreduce.jobtracker.address
2015-02-12 16:12:54,005 INFO  mapreduce.JobSubmitter (JobSubmitter.java:printTokens(575)) - Submitting tokens for job: job_local155775041_0035
2015-02-12 16:12:54,016 WARN  conf.Configuration (Configuration.java:loadProperty(2510)) - file:/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/tmp/cascading/staging/soberoi155775041/.staging/job_local155775041_0035/job.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.retry.interval;  Ignoring.
2015-02-12 16:12:54,016 WARN  conf.Configuration (Configuration.java:loadProperty(2510)) - file:/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/tmp/cascading/staging/soberoi155775041/.staging/job_local155775041_0035/job.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.attempts;  Ignoring.
2015-02-12 16:12:54,045 WARN  conf.Configuration (Configuration.java:loadProperty(2510)) - file:/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/test/tmp/mapred/local/localRunner/soberoi/job_local155775041_0035/job_local155775041_0035.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.retry.interval;  Ignoring.
2015-02-12 16:12:54,046 WARN  conf.Configuration (Configuration.java:loadProperty(2510)) - file:/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/test/tmp/mapred/local/localRunner/soberoi/job_local155775041_0035/job_local155775041_0035.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.attempts;  Ignoring.
2015-02-12 16:12:54,046 INFO  mapreduce.Job (Job.java:submit(1299)) - The url to track the job: http://localhost:8080/
2015-02-12 16:12:54,046 INFO  mapred.LocalJobRunner (LocalJobRunner.java:createOutputCommitter(471)) - OutputCommitter set in config null
2015-02-12 16:12:54,047 INFO  flow.FlowStep (BaseFlowStep.java:logInfo(834)) - [count-lhs*source*count...] submitted hadoop job: job_local155775041_0035
2015-02-12 16:12:54,047 INFO  mapred.LocalJobRunner (LocalJobRunner.java:createOutputCommitter(489)) - OutputCommitter is org.apache.hadoop.mapred.FileOutputCommitter
2015-02-12 16:12:54,047 INFO  flow.FlowStep (BaseFlowStep.java:logInfo(834)) - [count-lhs*source*count...] tracking url: http://localhost:8080/
2015-02-12 16:12:54,049 INFO  mapred.LocalJobRunner (LocalJobRunner.java:runTasks(448)) - Waiting for map tasks
2015-02-12 16:12:54,049 INFO  mapred.LocalJobRunner (LocalJobRunner.java:run(224)) - Starting task: attempt_local155775041_0035_m_000000_0
2015-02-12 16:12:54,050 INFO  util.ProcfsBasedProcessTree (ProcfsBasedProcessTree.java:isAvailable(181)) - ProcfsBasedProcessTree currently is supported only on Linux.
2015-02-12 16:12:54,050 INFO  mapred.Task (Task.java:initialize(587)) -  Using ResourceCalculatorProcessTree : null
2015-02-12 16:12:54,051 INFO  io.MultiInputSplit (MultiInputSplit.java:readFields(161)) - current split input path: file:/tmp/hadoop-soberoi/3798598062_source_BF20C2B770DC46CE8090B9EEF941B213/part-00000
2015-02-12 16:12:54,051 INFO  mapred.MapTask (MapTask.java:updateJobWithSplit(482)) - Processing split: cascading.tap.hadoop.io.MultiInputSplit@26ef9cf5
2015-02-12 16:12:54,052 INFO  mapred.MapTask (MapTask.java:runOldMapper(436)) - numReduceTasks: 1
2015-02-12 16:12:54,067 INFO  mapred.MapTask (MapTask.java:setEquator(1202)) - (EQUATOR) 0 kvi 26214396(104857584)
2015-02-12 16:12:54,067 INFO  mapred.MapTask (MapTask.java:init(995)) - mapreduce.task.io.sort.mb: 100
2015-02-12 16:12:54,068 INFO  mapred.MapTask (MapTask.java:init(996)) - soft limit at 83886080
2015-02-12 16:12:54,068 INFO  mapred.MapTask (MapTask.java:init(997)) - bufstart = 0; bufvoid = 104857600
2015-02-12 16:12:54,068 INFO  mapred.MapTask (MapTask.java:init(998)) - kvstart = 26214396; length = 6553600
2015-02-12 16:12:54,069 INFO  mapred.MapTask (MapTask.java:createSortingCollector(402)) - Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer
2015-02-12 16:12:54,069 INFO  hadoop.FlowMapper (FlowMapper.java:configure(71)) - cascading version: 2.6.3
2015-02-12 16:12:54,069 INFO  hadoop.FlowMapper (FlowMapper.java:configure(72)) - child jvm opts: -Xmx200m
2015-02-12 16:12:54,072 INFO  hadoop.FlowMapper (FlowMapper.java:configure(87)) - sourcing from: TempHfs["SequenceFile[['char', 'count']]"][3798598062/source/]
2015-02-12 16:12:54,073 INFO  hadoop.FlowMapper (FlowMapper.java:configure(90)) - sinking to: CoGroup(source*count-rhs)[by: source:[{1}:'char'] count-rhs:[{1}:'char']]
2015-02-12 16:12:54,074 INFO  mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) - 
2015-02-12 16:12:54,074 INFO  mapred.MapTask (MapTask.java:flush(1457)) - Starting flush of map output
2015-02-12 16:12:54,074 INFO  mapred.MapTask (MapTask.java:flush(1475)) - Spilling map output
2015-02-12 16:12:54,074 INFO  mapred.MapTask (MapTask.java:flush(1476)) - bufstart = 0; bufend = 65; bufvoid = 104857600
2015-02-12 16:12:54,074 INFO  mapred.MapTask (MapTask.java:flush(1478)) - kvstart = 26214396(104857584); kvend = 26214380(104857520); length = 17/6553600
2015-02-12 16:12:54,075 INFO  mapred.MapTask (MapTask.java:sortAndSpill(1660)) - Finished spill 0
2015-02-12 16:12:54,077 INFO  mapred.Task (Task.java:done(1001)) - Task:attempt_local155775041_0035_m_000000_0 is done. And is in the process of committing
2015-02-12 16:12:54,077 INFO  mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) - file:/tmp/hadoop-soberoi/3798598062_source_BF20C2B770DC46CE8090B9EEF941B213/part-00000:0+160
2015-02-12 16:12:54,077 INFO  mapred.Task (Task.java:sendDone(1121)) - Task 'attempt_local155775041_0035_m_000000_0' done.
2015-02-12 16:12:54,078 INFO  mapred.LocalJobRunner (LocalJobRunner.java:run(249)) - Finishing task: attempt_local155775041_0035_m_000000_0
2015-02-12 16:12:54,078 INFO  mapred.LocalJobRunner (LocalJobRunner.java:run(224)) - Starting task: attempt_local155775041_0035_m_000001_0
2015-02-12 16:12:54,078 INFO  util.ProcfsBasedProcessTree (ProcfsBasedProcessTree.java:isAvailable(181)) - ProcfsBasedProcessTree currently is supported only on Linux.
2015-02-12 16:12:54,079 INFO  mapred.Task (Task.java:initialize(587)) -  Using ResourceCalculatorProcessTree : null
2015-02-12 16:12:54,079 INFO  io.MultiInputSplit (MultiInputSplit.java:readFields(161)) - current split input path: file:/tmp/hadoop-soberoi/3761761564_count_rhs_F5332AAA9616477EAC6E92C77907B8AC/part-00000
2015-02-12 16:12:54,079 INFO  mapred.MapTask (MapTask.java:updateJobWithSplit(482)) - Processing split: cascading.tap.hadoop.io.MultiInputSplit@3f078f1c
2015-02-12 16:12:54,080 INFO  mapred.MapTask (MapTask.java:runOldMapper(436)) - numReduceTasks: 1
2015-02-12 16:12:54,091 INFO  mapred.MapTask (MapTask.java:setEquator(1202)) - (EQUATOR) 0 kvi 26214396(104857584)
2015-02-12 16:12:54,092 INFO  mapred.MapTask (MapTask.java:init(995)) - mapreduce.task.io.sort.mb: 100
2015-02-12 16:12:54,092 INFO  mapred.MapTask (MapTask.java:init(996)) - soft limit at 83886080
2015-02-12 16:12:54,092 INFO  mapred.MapTask (MapTask.java:init(997)) - bufstart = 0; bufvoid = 104857600
2015-02-12 16:12:54,092 INFO  mapred.MapTask (MapTask.java:init(998)) - kvstart = 26214396; length = 6553600
2015-02-12 16:12:54,092 INFO  mapred.MapTask (MapTask.java:createSortingCollector(402)) - Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer
2015-02-12 16:12:54,093 INFO  hadoop.FlowMapper (FlowMapper.java:configure(71)) - cascading version: 2.6.3
2015-02-12 16:12:54,093 INFO  hadoop.FlowMapper (FlowMapper.java:configure(72)) - child jvm opts: -Xmx200m
2015-02-12 16:12:54,095 INFO  hadoop.FlowMapper (FlowMapper.java:configure(87)) - sourcing from: TempHfs["SequenceFile[['char', 'count']]"][3761761564/count-rhs/]
2015-02-12 16:12:54,095 INFO  hadoop.FlowMapper (FlowMapper.java:configure(90)) - sinking to: CoGroup(source*count-rhs)[by: source:[{1}:'char'] count-rhs:[{1}:'char']]
2015-02-12 16:12:54,096 INFO  mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) - 
2015-02-12 16:12:54,097 INFO  mapred.MapTask (MapTask.java:flush(1457)) - Starting flush of map output
2015-02-12 16:12:54,097 INFO  mapred.MapTask (MapTask.java:flush(1475)) - Spilling map output
2015-02-12 16:12:54,097 INFO  mapred.MapTask (MapTask.java:flush(1476)) - bufstart = 0; bufend = 65; bufvoid = 104857600
2015-02-12 16:12:54,097 INFO  mapred.MapTask (MapTask.java:flush(1478)) - kvstart = 26214396(104857584); kvend = 26214380(104857520); length = 17/6553600
2015-02-12 16:12:54,098 INFO  mapred.MapTask (MapTask.java:sortAndSpill(1660)) - Finished spill 0
2015-02-12 16:12:54,100 INFO  mapred.Task (Task.java:done(1001)) - Task:attempt_local155775041_0035_m_000001_0 is done. And is in the process of committing
2015-02-12 16:12:54,101 INFO  mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) - file:/tmp/hadoop-soberoi/3761761564_count_rhs_F5332AAA9616477EAC6E92C77907B8AC/part-00000:0+160
2015-02-12 16:12:54,101 INFO  mapred.Task (Task.java:sendDone(1121)) - Task 'attempt_local155775041_0035_m_000001_0' done.
2015-02-12 16:12:54,101 INFO  mapred.LocalJobRunner (LocalJobRunner.java:run(249)) - Finishing task: attempt_local155775041_0035_m_000001_0
2015-02-12 16:12:54,101 INFO  mapred.LocalJobRunner (LocalJobRunner.java:runTasks(456)) - map task executor complete.
2015-02-12 16:12:54,101 INFO  mapred.LocalJobRunner (LocalJobRunner.java:runTasks(448)) - Waiting for reduce tasks
2015-02-12 16:12:54,102 INFO  mapred.LocalJobRunner (LocalJobRunner.java:run(302)) - Starting task: attempt_local155775041_0035_r_000000_0
2015-02-12 16:12:54,102 INFO  util.ProcfsBasedProcessTree (ProcfsBasedProcessTree.java:isAvailable(181)) - ProcfsBasedProcessTree currently is supported only on Linux.
2015-02-12 16:12:54,103 INFO  mapred.Task (Task.java:initialize(587)) -  Using ResourceCalculatorProcessTree : null
2015-02-12 16:12:54,103 INFO  mapred.ReduceTask (ReduceTask.java:run(362)) - Using ShuffleConsumerPlugin: org.apache.hadoop.mapreduce.task.reduce.Shuffle@38765d
2015-02-12 16:12:54,103 INFO  reduce.MergeManagerImpl (MergeManagerImpl.java:<init>(195)) - MergerManager: memoryLimit=493250144, maxSingleShuffleLimit=123312536, mergeThreshold=325545120, ioSortFactor=10, memToMemMergeOutputsThreshold=10
2015-02-12 16:12:54,104 INFO  reduce.EventFetcher (EventFetcher.java:run(61)) - attempt_local155775041_0035_r_000000_0 Thread started: EventFetcher for fetching Map Completion Events
2015-02-12 16:12:54,105 INFO  reduce.LocalFetcher (LocalFetcher.java:copyMapOutput(141)) - localfetcher#28 about to shuffle output of map attempt_local155775041_0035_m_000000_0 decomp: 77 len: 81 to MEMORY
2015-02-12 16:12:54,105 INFO  reduce.InMemoryMapOutput (InMemoryMapOutput.java:shuffle(100)) - Read 77 bytes from map-output for attempt_local155775041_0035_m_000000_0
2015-02-12 16:12:54,105 INFO  reduce.MergeManagerImpl (MergeManagerImpl.java:closeInMemoryFile(313)) - closeInMemoryFile -> map-output of size: 77, inMemoryMapOutputs.size() -> 1, commitMemory -> 0, usedMemory ->77
2015-02-12 16:12:54,106 INFO  reduce.LocalFetcher (LocalFetcher.java:copyMapOutput(141)) - localfetcher#28 about to shuffle output of map attempt_local155775041_0035_m_000001_0 decomp: 77 len: 81 to MEMORY
2015-02-12 16:12:54,107 INFO  reduce.InMemoryMapOutput (InMemoryMapOutput.java:shuffle(100)) - Read 77 bytes from map-output for attempt_local155775041_0035_m_000001_0
2015-02-12 16:12:54,107 INFO  reduce.MergeManagerImpl (MergeManagerImpl.java:closeInMemoryFile(313)) - closeInMemoryFile -> map-output of size: 77, inMemoryMapOutputs.size() -> 2, commitMemory -> 77, usedMemory ->154
2015-02-12 16:12:54,107 INFO  reduce.EventFetcher (EventFetcher.java:run(76)) - EventFetcher is interrupted.. Returning
2015-02-12 16:12:54,107 INFO  mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) - 2 / 2 copied.
2015-02-12 16:12:54,108 INFO  reduce.MergeManagerImpl (MergeManagerImpl.java:finalMerge(673)) - finalMerge called with 2 in-memory map-outputs and 0 on-disk map-outputs
2015-02-12 16:12:54,109 INFO  mapred.Merger (Merger.java:merge(606)) - Merging 2 sorted segments
2015-02-12 16:12:54,109 INFO  mapred.Merger (Merger.java:merge(705)) - Down to the last merge-pass, with 2 segments left of total size: 134 bytes
2015-02-12 16:12:54,111 INFO  reduce.MergeManagerImpl (MergeManagerImpl.java:finalMerge(750)) - Merged 2 segments, 154 bytes to disk to satisfy reduce memory limit
2015-02-12 16:12:54,111 INFO  reduce.MergeManagerImpl (MergeManagerImpl.java:finalMerge(780)) - Merging 1 files, 156 bytes from disk
2015-02-12 16:12:54,111 INFO  reduce.MergeManagerImpl (MergeManagerImpl.java:finalMerge(795)) - Merging 0 segments, 0 bytes from memory into reduce
2015-02-12 16:12:54,111 INFO  mapred.Merger (Merger.java:merge(606)) - Merging 1 sorted segments
2015-02-12 16:12:54,111 INFO  mapred.Merger (Merger.java:merge(705)) - Down to the last merge-pass, with 1 segments left of total size: 142 bytes
2015-02-12 16:12:54,112 INFO  mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) - 2 / 2 copied.
2015-02-12 16:12:54,112 INFO  hadoop.FlowReducer (FlowReducer.java:configure(78)) - cascading version: 2.6.3
2015-02-12 16:12:54,112 INFO  hadoop.FlowReducer (FlowReducer.java:configure(79)) - child jvm opts: -Xmx200m
2015-02-12 16:12:54,115 INFO  hadoop.FlowReducer (FlowReducer.java:configure(97)) - sourcing from: CoGroup(source*count-rhs)[by: source:[{1}:'char'] count-rhs:[{1}:'char']]
2015-02-12 16:12:54,115 INFO  hadoop.FlowReducer (FlowReducer.java:configure(100)) - sinking to: TempHfs["SequenceFile[['char', 'num', 'char2', 'count']]"][6501264219/source_count-rhs/]
2015-02-12 16:12:54,125 INFO  collect.SpillableTupleList (SpillableTupleList.java:getCodecClass(105)) - attempting to load codec: org.apache.hadoop.io.compress.GzipCodec
2015-02-12 16:12:54,125 INFO  collect.SpillableTupleList (SpillableTupleList.java:getCodecClass(110)) - found codec: org.apache.hadoop.io.compress.GzipCodec
2015-02-12 16:12:54,129 INFO  mapred.Task (Task.java:done(1001)) - Task:attempt_local155775041_0035_r_000000_0 is done. And is in the process of committing
2015-02-12 16:12:54,130 INFO  mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) - 2 / 2 copied.
2015-02-12 16:12:54,130 INFO  mapred.Task (Task.java:commit(1162)) - Task attempt_local155775041_0035_r_000000_0 is allowed to commit now
2015-02-12 16:12:54,130 INFO  output.FileOutputCommitter (FileOutputCommitter.java:commitTask(439)) - Saved output of task 'attempt_local155775041_0035_r_000000_0' to file:/tmp/hadoop-soberoi/6501264219_source_count_rhs_BC66FC7556A146A5A30829F37762504A/_temporary/0/task_local155775041_0035_r_000000
2015-02-12 16:12:54,131 INFO  mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) - reduce > reduce
2015-02-12 16:12:54,131 INFO  mapred.Task (Task.java:sendDone(1121)) - Task 'attempt_local155775041_0035_r_000000_0' done.
2015-02-12 16:12:54,131 INFO  mapred.LocalJobRunner (LocalJobRunner.java:run(325)) - Finishing task: attempt_local155775041_0035_r_000000_0
2015-02-12 16:12:54,131 INFO  mapred.LocalJobRunner (LocalJobRunner.java:runTasks(456)) - reduce task executor complete.
2015-02-12 16:12:54,143 INFO  flow.FlowStep (BaseFlowStep.java:logInfo(834)) - [count-lhs*source*count...] starting step: (5/5) ...m/samesourcemergethreeway
2015-02-12 16:12:54,144 INFO  jvm.JvmMetrics (JvmMetrics.java:init(71)) - Cannot initialize JVM Metrics with processName=JobTracker, sessionId= - already initialized
2015-02-12 16:12:54,145 INFO  jvm.JvmMetrics (JvmMetrics.java:init(71)) - Cannot initialize JVM Metrics with processName=JobTracker, sessionId= - already initialized
2015-02-12 16:12:54,177 INFO  mapred.FileInputFormat (FileInputFormat.java:listStatus(247)) - Total input paths to process : 1
2015-02-12 16:12:54,188 INFO  mapred.FileInputFormat (FileInputFormat.java:listStatus(247)) - Total input paths to process : 1
2015-02-12 16:12:54,205 INFO  mapreduce.JobSubmitter (JobSubmitter.java:submitJobInternal(492)) - number of splits:2
2015-02-12 16:12:54,214 INFO  Configuration.deprecation (Configuration.java:warnOnceIfDeprecated(1022)) - mapred.used.genericoptionsparser is deprecated. Instead, use mapreduce.client.genericoptionsparser.used
2015-02-12 16:12:54,214 INFO  Configuration.deprecation (Configuration.java:warnOnceIfDeprecated(1022)) - mapred.job.tracker is deprecated. Instead, use mapreduce.jobtracker.address
2015-02-12 16:12:54,217 INFO  mapreduce.JobSubmitter (JobSubmitter.java:printTokens(575)) - Submitting tokens for job: job_local1087895190_0036
2015-02-12 16:12:54,227 WARN  conf.Configuration (Configuration.java:loadProperty(2510)) - file:/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/tmp/cascading/staging/soberoi1087895190/.staging/job_local1087895190_0036/job.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.retry.interval;  Ignoring.
2015-02-12 16:12:54,227 WARN  conf.Configuration (Configuration.java:loadProperty(2510)) - file:/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/tmp/cascading/staging/soberoi1087895190/.staging/job_local1087895190_0036/job.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.attempts;  Ignoring.
2015-02-12 16:12:54,261 WARN  conf.Configuration (Configuration.java:loadProperty(2510)) - file:/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/test/tmp/mapred/local/localRunner/soberoi/job_local1087895190_0036/job_local1087895190_0036.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.retry.interval;  Ignoring.
2015-02-12 16:12:54,262 WARN  conf.Configuration (Configuration.java:loadProperty(2510)) - file:/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/test/tmp/mapred/local/localRunner/soberoi/job_local1087895190_0036/job_local1087895190_0036.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.attempts;  Ignoring.
2015-02-12 16:12:54,262 INFO  mapreduce.Job (Job.java:submit(1299)) - The url to track the job: http://localhost:8080/
2015-02-12 16:12:54,263 INFO  mapred.LocalJobRunner (LocalJobRunner.java:createOutputCommitter(471)) - OutputCommitter set in config null
2015-02-12 16:12:54,263 INFO  flow.FlowStep (BaseFlowStep.java:logInfo(834)) - [count-lhs*source*count...] submitted hadoop job: job_local1087895190_0036
2015-02-12 16:12:54,263 INFO  mapred.LocalJobRunner (LocalJobRunner.java:createOutputCommitter(489)) - OutputCommitter is org.apache.hadoop.mapred.FileOutputCommitter
2015-02-12 16:12:54,263 INFO  flow.FlowStep (BaseFlowStep.java:logInfo(834)) - [count-lhs*source*count...] tracking url: http://localhost:8080/
2015-02-12 16:12:54,265 INFO  mapred.LocalJobRunner (LocalJobRunner.java:runTasks(448)) - Waiting for map tasks
2015-02-12 16:12:54,265 INFO  mapred.LocalJobRunner (LocalJobRunner.java:run(224)) - Starting task: attempt_local1087895190_0036_m_000000_0
2015-02-12 16:12:54,266 INFO  util.ProcfsBasedProcessTree (ProcfsBasedProcessTree.java:isAvailable(181)) - ProcfsBasedProcessTree currently is supported only on Linux.
2015-02-12 16:12:54,266 INFO  mapred.Task (Task.java:initialize(587)) -  Using ResourceCalculatorProcessTree : null
2015-02-12 16:12:54,266 INFO  io.MultiInputSplit (MultiInputSplit.java:readFields(161)) - current split input path: file:/tmp/hadoop-soberoi/6501264219_source_count_rhs_BC66FC7556A146A5A30829F37762504A/part-00000
2015-02-12 16:12:54,267 INFO  mapred.MapTask (MapTask.java:updateJobWithSplit(482)) - Processing split: cascading.tap.hadoop.io.MultiInputSplit@39d47789
2015-02-12 16:12:54,267 INFO  mapred.MapTask (MapTask.java:runOldMapper(436)) - numReduceTasks: 1
2015-02-12 16:12:54,278 INFO  mapred.MapTask (MapTask.java:setEquator(1202)) - (EQUATOR) 0 kvi 26214396(104857584)
2015-02-12 16:12:54,278 INFO  mapred.MapTask (MapTask.java:init(995)) - mapreduce.task.io.sort.mb: 100
2015-02-12 16:12:54,278 INFO  mapred.MapTask (MapTask.java:init(996)) - soft limit at 83886080
2015-02-12 16:12:54,278 INFO  mapred.MapTask (MapTask.java:init(997)) - bufstart = 0; bufvoid = 104857600
2015-02-12 16:12:54,278 INFO  mapred.MapTask (MapTask.java:init(998)) - kvstart = 26214396; length = 6553600
2015-02-12 16:12:54,279 INFO  mapred.MapTask (MapTask.java:createSortingCollector(402)) - Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer
2015-02-12 16:12:54,279 INFO  hadoop.FlowMapper (FlowMapper.java:configure(71)) - cascading version: 2.6.3
2015-02-12 16:12:54,279 INFO  hadoop.FlowMapper (FlowMapper.java:configure(72)) - child jvm opts: -Xmx200m
2015-02-12 16:12:54,283 INFO  hadoop.FlowMapper (FlowMapper.java:configure(87)) - sourcing from: TempHfs["SequenceFile[['char', 'num', 'char2', 'count']]"][6501264219/source_count-rhs/]
2015-02-12 16:12:54,283 INFO  hadoop.FlowMapper (FlowMapper.java:configure(90)) - sinking to: CoGroup(count-lhs*source*count-rhs)[by: count-lhs:[{1}:'char'] source*count-rhs:[{1}:'char']]
2015-02-12 16:12:54,284 INFO  mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) - 
2015-02-12 16:12:54,284 INFO  mapred.MapTask (MapTask.java:flush(1457)) - Starting flush of map output
2015-02-12 16:12:54,284 INFO  mapred.MapTask (MapTask.java:flush(1475)) - Spilling map output
2015-02-12 16:12:54,284 INFO  mapred.MapTask (MapTask.java:flush(1476)) - bufstart = 0; bufend = 105; bufvoid = 104857600
2015-02-12 16:12:54,284 INFO  mapred.MapTask (MapTask.java:flush(1478)) - kvstart = 26214396(104857584); kvend = 26214380(104857520); length = 17/6553600
2015-02-12 16:12:54,286 INFO  mapred.MapTask (MapTask.java:sortAndSpill(1660)) - Finished spill 0
2015-02-12 16:12:54,287 INFO  mapred.Task (Task.java:done(1001)) - Task:attempt_local1087895190_0036_m_000000_0 is done. And is in the process of committing
2015-02-12 16:12:54,288 INFO  mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) - file:/tmp/hadoop-soberoi/6501264219_source_count_rhs_BC66FC7556A146A5A30829F37762504A/part-00000:0+200
2015-02-12 16:12:54,288 INFO  mapred.Task (Task.java:sendDone(1121)) - Task 'attempt_local1087895190_0036_m_000000_0' done.
2015-02-12 16:12:54,288 INFO  mapred.LocalJobRunner (LocalJobRunner.java:run(249)) - Finishing task: attempt_local1087895190_0036_m_000000_0
2015-02-12 16:12:54,288 INFO  mapred.LocalJobRunner (LocalJobRunner.java:run(224)) - Starting task: attempt_local1087895190_0036_m_000001_0
2015-02-12 16:12:54,289 INFO  util.ProcfsBasedProcessTree (ProcfsBasedProcessTree.java:isAvailable(181)) - ProcfsBasedProcessTree currently is supported only on Linux.
2015-02-12 16:12:54,289 INFO  mapred.Task (Task.java:initialize(587)) -  Using ResourceCalculatorProcessTree : null
2015-02-12 16:12:54,290 INFO  io.MultiInputSplit (MultiInputSplit.java:readFields(161)) - current split input path: file:/tmp/hadoop-soberoi/6668818197_count_lhs_E0299A91C9AD4B61A1E657A2086B2F4E/part-00000
2015-02-12 16:12:54,290 INFO  mapred.MapTask (MapTask.java:updateJobWithSplit(482)) - Processing split: cascading.tap.hadoop.io.MultiInputSplit@639f8c4a
2015-02-12 16:12:54,291 INFO  mapred.MapTask (MapTask.java:runOldMapper(436)) - numReduceTasks: 1
2015-02-12 16:12:54,302 INFO  mapred.MapTask (MapTask.java:setEquator(1202)) - (EQUATOR) 0 kvi 26214396(104857584)
2015-02-12 16:12:54,302 INFO  mapred.MapTask (MapTask.java:init(995)) - mapreduce.task.io.sort.mb: 100
2015-02-12 16:12:54,302 INFO  mapred.MapTask (MapTask.java:init(996)) - soft limit at 83886080
2015-02-12 16:12:54,302 INFO  mapred.MapTask (MapTask.java:init(997)) - bufstart = 0; bufvoid = 104857600
2015-02-12 16:12:54,302 INFO  mapred.MapTask (MapTask.java:init(998)) - kvstart = 26214396; length = 6553600
2015-02-12 16:12:54,303 INFO  mapred.MapTask (MapTask.java:createSortingCollector(402)) - Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer
2015-02-12 16:12:54,303 INFO  hadoop.FlowMapper (FlowMapper.java:configure(71)) - cascading version: 2.6.3
2015-02-12 16:12:54,303 INFO  hadoop.FlowMapper (FlowMapper.java:configure(72)) - child jvm opts: -Xmx200m
2015-02-12 16:12:54,306 INFO  hadoop.FlowMapper (FlowMapper.java:configure(87)) - sourcing from: TempHfs["SequenceFile[['char', 'count']]"][6668818197/count-lhs/]
2015-02-12 16:12:54,306 INFO  hadoop.FlowMapper (FlowMapper.java:configure(90)) - sinking to: CoGroup(count-lhs*source*count-rhs)[by: count-lhs:[{1}:'char'] source*count-rhs:[{1}:'char']]
2015-02-12 16:12:54,308 INFO  mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) - 
2015-02-12 16:12:54,308 INFO  mapred.MapTask (MapTask.java:flush(1457)) - Starting flush of map output
2015-02-12 16:12:54,308 INFO  mapred.MapTask (MapTask.java:flush(1475)) - Spilling map output
2015-02-12 16:12:54,308 INFO  mapred.MapTask (MapTask.java:flush(1476)) - bufstart = 0; bufend = 65; bufvoid = 104857600
2015-02-12 16:12:54,308 INFO  mapred.MapTask (MapTask.java:flush(1478)) - kvstart = 26214396(104857584); kvend = 26214380(104857520); length = 17/6553600
2015-02-12 16:12:54,309 INFO  mapred.MapTask (MapTask.java:sortAndSpill(1660)) - Finished spill 0
2015-02-12 16:12:54,310 INFO  mapred.Task (Task.java:done(1001)) - Task:attempt_local1087895190_0036_m_000001_0 is done. And is in the process of committing
2015-02-12 16:12:54,311 INFO  mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) - file:/tmp/hadoop-soberoi/6668818197_count_lhs_E0299A91C9AD4B61A1E657A2086B2F4E/part-00000:0+160
2015-02-12 16:12:54,311 INFO  mapred.Task (Task.java:sendDone(1121)) - Task 'attempt_local1087895190_0036_m_000001_0' done.
2015-02-12 16:12:54,311 INFO  mapred.LocalJobRunner (LocalJobRunner.java:run(249)) - Finishing task: attempt_local1087895190_0036_m_000001_0
2015-02-12 16:12:54,311 INFO  mapred.LocalJobRunner (LocalJobRunner.java:runTasks(456)) - map task executor complete.
2015-02-12 16:12:54,312 INFO  mapred.LocalJobRunner (LocalJobRunner.java:runTasks(448)) - Waiting for reduce tasks
2015-02-12 16:12:54,312 INFO  mapred.LocalJobRunner (LocalJobRunner.java:run(302)) - Starting task: attempt_local1087895190_0036_r_000000_0
2015-02-12 16:12:54,313 INFO  util.ProcfsBasedProcessTree (ProcfsBasedProcessTree.java:isAvailable(181)) - ProcfsBasedProcessTree currently is supported only on Linux.
2015-02-12 16:12:54,313 INFO  mapred.Task (Task.java:initialize(587)) -  Using ResourceCalculatorProcessTree : null
2015-02-12 16:12:54,313 INFO  mapred.ReduceTask (ReduceTask.java:run(362)) - Using ShuffleConsumerPlugin: org.apache.hadoop.mapreduce.task.reduce.Shuffle@570fd91e
2015-02-12 16:12:54,314 INFO  reduce.MergeManagerImpl (MergeManagerImpl.java:<init>(195)) - MergerManager: memoryLimit=493250144, maxSingleShuffleLimit=123312536, mergeThreshold=325545120, ioSortFactor=10, memToMemMergeOutputsThreshold=10
2015-02-12 16:12:54,315 INFO  reduce.EventFetcher (EventFetcher.java:run(61)) - attempt_local1087895190_0036_r_000000_0 Thread started: EventFetcher for fetching Map Completion Events
2015-02-12 16:12:54,316 INFO  reduce.LocalFetcher (LocalFetcher.java:copyMapOutput(141)) - localfetcher#29 about to shuffle output of map attempt_local1087895190_0036_m_000001_0 decomp: 77 len: 81 to MEMORY
2015-02-12 16:12:54,316 INFO  reduce.InMemoryMapOutput (InMemoryMapOutput.java:shuffle(100)) - Read 77 bytes from map-output for attempt_local1087895190_0036_m_000001_0
2015-02-12 16:12:54,316 INFO  reduce.MergeManagerImpl (MergeManagerImpl.java:closeInMemoryFile(313)) - closeInMemoryFile -> map-output of size: 77, inMemoryMapOutputs.size() -> 1, commitMemory -> 0, usedMemory ->77
2015-02-12 16:12:54,317 INFO  reduce.LocalFetcher (LocalFetcher.java:copyMapOutput(141)) - localfetcher#29 about to shuffle output of map attempt_local1087895190_0036_m_000000_0 decomp: 117 len: 121 to MEMORY
2015-02-12 16:12:54,317 INFO  reduce.InMemoryMapOutput (InMemoryMapOutput.java:shuffle(100)) - Read 117 bytes from map-output for attempt_local1087895190_0036_m_000000_0
2015-02-12 16:12:54,317 INFO  reduce.MergeManagerImpl (MergeManagerImpl.java:closeInMemoryFile(313)) - closeInMemoryFile -> map-output of size: 117, inMemoryMapOutputs.size() -> 2, commitMemory -> 77, usedMemory ->194
2015-02-12 16:12:54,317 INFO  reduce.EventFetcher (EventFetcher.java:run(76)) - EventFetcher is interrupted.. Returning
2015-02-12 16:12:54,318 INFO  mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) - 2 / 2 copied.
2015-02-12 16:12:54,318 INFO  reduce.MergeManagerImpl (MergeManagerImpl.java:finalMerge(673)) - finalMerge called with 2 in-memory map-outputs and 0 on-disk map-outputs
2015-02-12 16:12:54,319 INFO  mapred.Merger (Merger.java:merge(606)) - Merging 2 sorted segments
2015-02-12 16:12:54,319 INFO  mapred.Merger (Merger.java:merge(705)) - Down to the last merge-pass, with 2 segments left of total size: 174 bytes
2015-02-12 16:12:54,320 INFO  reduce.MergeManagerImpl (MergeManagerImpl.java:finalMerge(750)) - Merged 2 segments, 194 bytes to disk to satisfy reduce memory limit
2015-02-12 16:12:54,321 INFO  reduce.MergeManagerImpl (MergeManagerImpl.java:finalMerge(780)) - Merging 1 files, 196 bytes from disk
2015-02-12 16:12:54,321 INFO  reduce.MergeManagerImpl (MergeManagerImpl.java:finalMerge(795)) - Merging 0 segments, 0 bytes from memory into reduce
2015-02-12 16:12:54,321 INFO  mapred.Merger (Merger.java:merge(606)) - Merging 1 sorted segments
2015-02-12 16:12:54,321 INFO  mapred.Merger (Merger.java:merge(705)) - Down to the last merge-pass, with 1 segments left of total size: 182 bytes
2015-02-12 16:12:54,321 INFO  mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) - 2 / 2 copied.
2015-02-12 16:12:54,322 INFO  hadoop.FlowReducer (FlowReducer.java:configure(78)) - cascading version: 2.6.3
2015-02-12 16:12:54,322 INFO  hadoop.FlowReducer (FlowReducer.java:configure(79)) - child jvm opts: -Xmx200m
2015-02-12 16:12:54,325 INFO  hadoop.FlowReducer (FlowReducer.java:configure(97)) - sourcing from: CoGroup(count-lhs*source*count-rhs)[by: count-lhs:[{1}:'char'] source*count-rhs:[{1}:'char']]
2015-02-12 16:12:54,325 INFO  hadoop.FlowReducer (FlowReducer.java:configure(100)) - sinking to: Hfs["TextDelimited[['char', 'count']]"]["/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/test/output/hadoop2-mr1/assemblyhelpersplatform/samesourcemergethreeway"]
2015-02-12 16:12:54,331 INFO  collect.SpillableTupleList (SpillableTupleList.java:getCodecClass(105)) - attempting to load codec: org.apache.hadoop.io.compress.GzipCodec
2015-02-12 16:12:54,331 INFO  collect.SpillableTupleList (SpillableTupleList.java:getCodecClass(110)) - found codec: org.apache.hadoop.io.compress.GzipCodec
2015-02-12 16:12:54,333 INFO  mapred.Task (Task.java:done(1001)) - Task:attempt_local1087895190_0036_r_000000_0 is done. And is in the process of committing
2015-02-12 16:12:54,334 INFO  mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) - 2 / 2 copied.
2015-02-12 16:12:54,334 INFO  mapred.Task (Task.java:commit(1162)) - Task attempt_local1087895190_0036_r_000000_0 is allowed to commit now
2015-02-12 16:12:54,335 INFO  output.FileOutputCommitter (FileOutputCommitter.java:commitTask(439)) - Saved output of task 'attempt_local1087895190_0036_r_000000_0' to file:/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/test/output/hadoop2-mr1/assemblyhelpersplatform/samesourcemergethreeway/_temporary/0/task_local1087895190_0036_r_000000
2015-02-12 16:12:54,336 INFO  mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) - reduce > reduce
2015-02-12 16:12:54,336 INFO  mapred.Task (Task.java:sendDone(1121)) - Task 'attempt_local1087895190_0036_r_000000_0' done.
2015-02-12 16:12:54,336 INFO  mapred.LocalJobRunner (LocalJobRunner.java:run(325)) - Finishing task: attempt_local1087895190_0036_r_000000_0
2015-02-12 16:12:54,336 INFO  mapred.LocalJobRunner (LocalJobRunner.java:runTasks(456)) - reduce task executor complete.
2015-02-12 16:12:54,359 INFO  util.Hadoop18TapUtil (Hadoop18TapUtil.java:cleanTempPath(219)) - deleting temp path /Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/test/output/hadoop2-mr1/assemblyhelpersplatform/samesourcemergethreeway/_temporary
2015-02-12 16:12:54,378 INFO  mapred.FileInputFormat (FileInputFormat.java:listStatus(247)) - Total input paths to process : 1
2015-02-12 16:12:54,402 INFO  mapred.FileInputFormat (FileInputFormat.java:listStatus(247)) - Total input paths to process : 1
2015-02-12 16:12:54,403 INFO  cascading.PlatformTestCase (PlatformTestCase.java:tearDown(209)) - copying to local /Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/test/output//hadoop2-mr1/assemblyhelpersplatform/samesourcemergethreeway
2015-02-12 16:12:54,415 INFO  util.HadoopUtil (HadoopUtil.java:findMainClass(336)) - using default application jar, may cause class not found exceptions on the cluster
2015-02-12 16:12:54,415 INFO  planner.HadoopPlanner (HadoopPlanner.java:initialize(225)) - using application jar: /Users/soberoi/.gradle/caches/modules-2/files-2.1/cascading/cascading-hadoop2-mr1/2.6.3/9ae016e2b8d246b9c1de9168e715d9f1b971aa4a/cascading-hadoop2-mr1-2.6.3.jar
2015-02-12 16:12:54,419 INFO  Configuration.deprecation (Configuration.java:warnOnceIfDeprecated(1022)) - mapred.used.genericoptionsparser is deprecated. Instead, use mapreduce.client.genericoptionsparser.used
2015-02-12 16:12:54,420 INFO  Configuration.deprecation (Configuration.java:warnOnceIfDeprecated(1022)) - mapred.job.tracker is deprecated. Instead, use mapreduce.jobtracker.address
2015-02-12 16:12:54,420 INFO  Configuration.deprecation (Configuration.java:warnOnceIfDeprecated(1022)) - mapred.output.compress is deprecated. Instead, use mapreduce.output.fileoutputformat.compress
2015-02-12 16:12:54,425 INFO  flow.Flow (BaseFlow.java:logInfo(1378)) - [lhs+rhs] starting
2015-02-12 16:12:54,426 INFO  flow.Flow (BaseFlow.java:logInfo(1378)) - [lhs+rhs]  source: Hfs["TextLine[['offset', 'line']->[ALL]]"]["/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/classes/test/data/rhs.txt"]
2015-02-12 16:12:54,426 INFO  flow.Flow (BaseFlow.java:logInfo(1378)) - [lhs+rhs]  source: Hfs["TextLine[['offset', 'line']->[ALL]]"]["/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/classes/test/data/lhs.txt"]
2015-02-12 16:12:54,426 INFO  flow.Flow (BaseFlow.java:logInfo(1378)) - [lhs+rhs]  sink: Hfs["TextLine[['item']->['num', 'char']]"]["/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/test/output/hadoop2-mr1/assemblyhelpersplatform/uniquemerge-nondeterministic"]
2015-02-12 16:12:54,426 INFO  flow.Flow (BaseFlow.java:logInfo(1378)) - [lhs+rhs]  parallel execution is enabled: false
2015-02-12 16:12:54,426 INFO  flow.Flow (BaseFlow.java:logInfo(1378)) - [lhs+rhs]  starting jobs: 1
2015-02-12 16:12:54,426 INFO  flow.Flow (BaseFlow.java:logInfo(1378)) - [lhs+rhs]  allocating threads: 1
2015-02-12 16:12:54,426 INFO  flow.FlowStep (BaseFlowStep.java:logInfo(834)) - [lhs+rhs] starting step: (1/1) ...quemerge-nondeterministic
2015-02-12 16:12:54,427 INFO  jvm.JvmMetrics (JvmMetrics.java:init(71)) - Cannot initialize JVM Metrics with processName=JobTracker, sessionId= - already initialized
2015-02-12 16:12:54,427 INFO  jvm.JvmMetrics (JvmMetrics.java:init(71)) - Cannot initialize JVM Metrics with processName=JobTracker, sessionId= - already initialized
2015-02-12 16:12:54,444 INFO  mapred.FileInputFormat (FileInputFormat.java:listStatus(247)) - Total input paths to process : 1
2015-02-12 16:12:54,445 INFO  mapred.FileInputFormat (FileInputFormat.java:listStatus(247)) - Total input paths to process : 1
2015-02-12 16:12:54,461 INFO  mapreduce.JobSubmitter (JobSubmitter.java:submitJobInternal(492)) - number of splits:2
2015-02-12 16:12:54,472 INFO  mapreduce.JobSubmitter (JobSubmitter.java:printTokens(575)) - Submitting tokens for job: job_local1683387041_0037
2015-02-12 16:12:54,481 WARN  conf.Configuration (Configuration.java:loadProperty(2510)) - file:/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/tmp/cascading/staging/soberoi1683387041/.staging/job_local1683387041_0037/job.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.retry.interval;  Ignoring.
2015-02-12 16:12:54,482 WARN  conf.Configuration (Configuration.java:loadProperty(2510)) - file:/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/tmp/cascading/staging/soberoi1683387041/.staging/job_local1683387041_0037/job.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.attempts;  Ignoring.
2015-02-12 16:12:54,508 WARN  conf.Configuration (Configuration.java:loadProperty(2510)) - file:/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/test/tmp/mapred/local/localRunner/soberoi/job_local1683387041_0037/job_local1683387041_0037.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.retry.interval;  Ignoring.
2015-02-12 16:12:54,509 WARN  conf.Configuration (Configuration.java:loadProperty(2510)) - file:/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/test/tmp/mapred/local/localRunner/soberoi/job_local1683387041_0037/job_local1683387041_0037.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.attempts;  Ignoring.
2015-02-12 16:12:54,510 INFO  mapreduce.Job (Job.java:submit(1299)) - The url to track the job: http://localhost:8080/
2015-02-12 16:12:54,510 INFO  flow.FlowStep (BaseFlowStep.java:logInfo(834)) - [lhs+rhs] submitted hadoop job: job_local1683387041_0037
2015-02-12 16:12:54,510 INFO  mapred.LocalJobRunner (LocalJobRunner.java:createOutputCommitter(471)) - OutputCommitter set in config null
2015-02-12 16:12:54,510 INFO  mapred.LocalJobRunner (LocalJobRunner.java:createOutputCommitter(489)) - OutputCommitter is org.apache.hadoop.mapred.FileOutputCommitter
2015-02-12 16:12:54,510 INFO  flow.FlowStep (BaseFlowStep.java:logInfo(834)) - [lhs+rhs] tracking url: http://localhost:8080/
2015-02-12 16:12:54,512 INFO  mapred.LocalJobRunner (LocalJobRunner.java:runTasks(448)) - Waiting for map tasks
2015-02-12 16:12:54,512 INFO  mapred.LocalJobRunner (LocalJobRunner.java:run(224)) - Starting task: attempt_local1683387041_0037_m_000000_0
2015-02-12 16:12:54,513 INFO  util.ProcfsBasedProcessTree (ProcfsBasedProcessTree.java:isAvailable(181)) - ProcfsBasedProcessTree currently is supported only on Linux.
2015-02-12 16:12:54,513 INFO  mapred.Task (Task.java:initialize(587)) -  Using ResourceCalculatorProcessTree : null
2015-02-12 16:12:54,513 INFO  io.MultiInputSplit (MultiInputSplit.java:readFields(161)) - current split input path: file:/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/classes/test/data/lhs.txt
2015-02-12 16:12:54,514 INFO  mapred.MapTask (MapTask.java:updateJobWithSplit(482)) - Processing split: cascading.tap.hadoop.io.MultiInputSplit@65b2b2f0
2015-02-12 16:12:54,514 INFO  mapred.MapTask (MapTask.java:runOldMapper(436)) - numReduceTasks: 1
2015-02-12 16:12:54,525 INFO  mapred.MapTask (MapTask.java:setEquator(1202)) - (EQUATOR) 0 kvi 26214396(104857584)
2015-02-12 16:12:54,525 INFO  mapred.MapTask (MapTask.java:init(995)) - mapreduce.task.io.sort.mb: 100
2015-02-12 16:12:54,525 INFO  mapred.MapTask (MapTask.java:init(996)) - soft limit at 83886080
2015-02-12 16:12:54,525 INFO  mapred.MapTask (MapTask.java:init(997)) - bufstart = 0; bufvoid = 104857600
2015-02-12 16:12:54,525 INFO  mapred.MapTask (MapTask.java:init(998)) - kvstart = 26214396; length = 6553600
2015-02-12 16:12:54,526 INFO  mapred.MapTask (MapTask.java:createSortingCollector(402)) - Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer
2015-02-12 16:12:54,526 INFO  hadoop.FlowMapper (FlowMapper.java:configure(71)) - cascading version: 2.6.3
2015-02-12 16:12:54,526 INFO  hadoop.FlowMapper (FlowMapper.java:configure(72)) - child jvm opts: -Xmx200m
2015-02-12 16:12:54,528 INFO  hadoop.FlowMapper (FlowMapper.java:configure(87)) - sourcing from: Hfs["TextLine[['offset', 'line']->[ALL]]"]["/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/classes/test/data/lhs.txt"]
2015-02-12 16:12:54,529 INFO  hadoop.FlowMapper (FlowMapper.java:configure(90)) - sinking to: GroupBy(lhs+rhs)[by: lhs:[{1}:'num'] rhs:[{1}:'num']]
2015-02-12 16:12:54,532 INFO  mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) - 
2015-02-12 16:12:54,532 INFO  mapred.MapTask (MapTask.java:flush(1457)) - Starting flush of map output
2015-02-12 16:12:54,532 INFO  mapred.MapTask (MapTask.java:flush(1475)) - Spilling map output
2015-02-12 16:12:54,532 INFO  mapred.MapTask (MapTask.java:flush(1476)) - bufstart = 0; bufend = 75; bufvoid = 104857600
2015-02-12 16:12:54,532 INFO  mapred.MapTask (MapTask.java:flush(1478)) - kvstart = 26214396(104857584); kvend = 26214380(104857520); length = 17/6553600
2015-02-12 16:12:54,534 INFO  mapred.MapTask (MapTask.java:sortAndSpill(1660)) - Finished spill 0
2015-02-12 16:12:54,535 INFO  mapred.Task (Task.java:done(1001)) - Task:attempt_local1683387041_0037_m_000000_0 is done. And is in the process of committing
2015-02-12 16:12:54,536 INFO  mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) - file:/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/classes/test/data/lhs.txt:0+52
2015-02-12 16:12:54,536 INFO  mapred.Task (Task.java:sendDone(1121)) - Task 'attempt_local1683387041_0037_m_000000_0' done.
2015-02-12 16:12:54,536 INFO  mapred.LocalJobRunner (LocalJobRunner.java:run(249)) - Finishing task: attempt_local1683387041_0037_m_000000_0
2015-02-12 16:12:54,536 INFO  mapred.LocalJobRunner (LocalJobRunner.java:run(224)) - Starting task: attempt_local1683387041_0037_m_000001_0
2015-02-12 16:12:54,537 INFO  util.ProcfsBasedProcessTree (ProcfsBasedProcessTree.java:isAvailable(181)) - ProcfsBasedProcessTree currently is supported only on Linux.
2015-02-12 16:12:54,537 INFO  mapred.Task (Task.java:initialize(587)) -  Using ResourceCalculatorProcessTree : null
2015-02-12 16:12:54,538 INFO  io.MultiInputSplit (MultiInputSplit.java:readFields(161)) - current split input path: file:/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/classes/test/data/rhs.txt
2015-02-12 16:12:54,538 INFO  mapred.MapTask (MapTask.java:updateJobWithSplit(482)) - Processing split: cascading.tap.hadoop.io.MultiInputSplit@574e6595
2015-02-12 16:12:54,539 INFO  mapred.MapTask (MapTask.java:runOldMapper(436)) - numReduceTasks: 1
2015-02-12 16:12:54,555 INFO  mapred.MapTask (MapTask.java:setEquator(1202)) - (EQUATOR) 0 kvi 26214396(104857584)
2015-02-12 16:12:54,555 INFO  mapred.MapTask (MapTask.java:init(995)) - mapreduce.task.io.sort.mb: 100
2015-02-12 16:12:54,555 INFO  mapred.MapTask (MapTask.java:init(996)) - soft limit at 83886080
2015-02-12 16:12:54,555 INFO  mapred.MapTask (MapTask.java:init(997)) - bufstart = 0; bufvoid = 104857600
2015-02-12 16:12:54,555 INFO  mapred.MapTask (MapTask.java:init(998)) - kvstart = 26214396; length = 6553600
2015-02-12 16:12:54,556 INFO  mapred.MapTask (MapTask.java:createSortingCollector(402)) - Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer
2015-02-12 16:12:54,556 INFO  hadoop.FlowMapper (FlowMapper.java:configure(71)) - cascading version: 2.6.3
2015-02-12 16:12:54,556 INFO  hadoop.FlowMapper (FlowMapper.java:configure(72)) - child jvm opts: -Xmx200m
2015-02-12 16:12:54,559 INFO  hadoop.FlowMapper (FlowMapper.java:configure(87)) - sourcing from: Hfs["TextLine[['offset', 'line']->[ALL]]"]["/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/classes/test/data/rhs.txt"]
2015-02-12 16:12:54,559 INFO  hadoop.FlowMapper (FlowMapper.java:configure(90)) - sinking to: GroupBy(lhs+rhs)[by: lhs:[{1}:'num'] rhs:[{1}:'num']]
2015-02-12 16:12:54,561 INFO  mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) - 
2015-02-12 16:12:54,561 INFO  mapred.MapTask (MapTask.java:flush(1457)) - Starting flush of map output
2015-02-12 16:12:54,561 INFO  mapred.MapTask (MapTask.java:flush(1475)) - Spilling map output
2015-02-12 16:12:54,561 INFO  mapred.MapTask (MapTask.java:flush(1476)) - bufstart = 0; bufend = 75; bufvoid = 104857600
2015-02-12 16:12:54,561 INFO  mapred.MapTask (MapTask.java:flush(1478)) - kvstart = 26214396(104857584); kvend = 26214380(104857520); length = 17/6553600
2015-02-12 16:12:54,563 INFO  mapred.MapTask (MapTask.java:sortAndSpill(1660)) - Finished spill 0
2015-02-12 16:12:54,565 INFO  mapred.Task (Task.java:done(1001)) - Task:attempt_local1683387041_0037_m_000001_0 is done. And is in the process of committing
2015-02-12 16:12:54,566 INFO  mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) - file:/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/classes/test/data/rhs.txt:0+51
2015-02-12 16:12:54,566 INFO  mapred.Task (Task.java:sendDone(1121)) - Task 'attempt_local1683387041_0037_m_000001_0' done.
2015-02-12 16:12:54,566 INFO  mapred.LocalJobRunner (LocalJobRunner.java:run(249)) - Finishing task: attempt_local1683387041_0037_m_000001_0
2015-02-12 16:12:54,567 INFO  mapred.LocalJobRunner (LocalJobRunner.java:runTasks(456)) - map task executor complete.
2015-02-12 16:12:54,567 INFO  mapred.LocalJobRunner (LocalJobRunner.java:runTasks(448)) - Waiting for reduce tasks
2015-02-12 16:12:54,567 INFO  mapred.LocalJobRunner (LocalJobRunner.java:run(302)) - Starting task: attempt_local1683387041_0037_r_000000_0
2015-02-12 16:12:54,568 INFO  util.ProcfsBasedProcessTree (ProcfsBasedProcessTree.java:isAvailable(181)) - ProcfsBasedProcessTree currently is supported only on Linux.
2015-02-12 16:12:54,569 INFO  mapred.Task (Task.java:initialize(587)) -  Using ResourceCalculatorProcessTree : null
2015-02-12 16:12:54,569 INFO  mapred.ReduceTask (ReduceTask.java:run(362)) - Using ShuffleConsumerPlugin: org.apache.hadoop.mapreduce.task.reduce.Shuffle@5fe382df
2015-02-12 16:12:54,569 INFO  reduce.MergeManagerImpl (MergeManagerImpl.java:<init>(195)) - MergerManager: memoryLimit=493250144, maxSingleShuffleLimit=123312536, mergeThreshold=325545120, ioSortFactor=10, memToMemMergeOutputsThreshold=10
2015-02-12 16:12:54,570 INFO  reduce.EventFetcher (EventFetcher.java:run(61)) - attempt_local1683387041_0037_r_000000_0 Thread started: EventFetcher for fetching Map Completion Events
2015-02-12 16:12:54,570 INFO  reduce.LocalFetcher (LocalFetcher.java:copyMapOutput(141)) - localfetcher#30 about to shuffle output of map attempt_local1683387041_0037_m_000001_0 decomp: 87 len: 91 to MEMORY
2015-02-12 16:12:54,571 INFO  reduce.InMemoryMapOutput (InMemoryMapOutput.java:shuffle(100)) - Read 87 bytes from map-output for attempt_local1683387041_0037_m_000001_0
2015-02-12 16:12:54,571 INFO  reduce.MergeManagerImpl (MergeManagerImpl.java:closeInMemoryFile(313)) - closeInMemoryFile -> map-output of size: 87, inMemoryMapOutputs.size() -> 1, commitMemory -> 0, usedMemory ->87
2015-02-12 16:12:54,572 INFO  reduce.LocalFetcher (LocalFetcher.java:copyMapOutput(141)) - localfetcher#30 about to shuffle output of map attempt_local1683387041_0037_m_000000_0 decomp: 87 len: 91 to MEMORY
2015-02-12 16:12:54,573 INFO  reduce.InMemoryMapOutput (InMemoryMapOutput.java:shuffle(100)) - Read 87 bytes from map-output for attempt_local1683387041_0037_m_000000_0
2015-02-12 16:12:54,573 INFO  reduce.MergeManagerImpl (MergeManagerImpl.java:closeInMemoryFile(313)) - closeInMemoryFile -> map-output of size: 87, inMemoryMapOutputs.size() -> 2, commitMemory -> 87, usedMemory ->174
2015-02-12 16:12:54,573 INFO  reduce.EventFetcher (EventFetcher.java:run(76)) - EventFetcher is interrupted.. Returning
2015-02-12 16:12:54,574 INFO  mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) - 2 / 2 copied.
2015-02-12 16:12:54,574 INFO  reduce.MergeManagerImpl (MergeManagerImpl.java:finalMerge(673)) - finalMerge called with 2 in-memory map-outputs and 0 on-disk map-outputs
2015-02-12 16:12:54,575 INFO  mapred.Merger (Merger.java:merge(606)) - Merging 2 sorted segments
2015-02-12 16:12:54,576 INFO  mapred.Merger (Merger.java:merge(705)) - Down to the last merge-pass, with 2 segments left of total size: 156 bytes
2015-02-12 16:12:54,577 INFO  reduce.MergeManagerImpl (MergeManagerImpl.java:finalMerge(750)) - Merged 2 segments, 174 bytes to disk to satisfy reduce memory limit
2015-02-12 16:12:54,577 INFO  reduce.MergeManagerImpl (MergeManagerImpl.java:finalMerge(780)) - Merging 1 files, 176 bytes from disk
2015-02-12 16:12:54,577 INFO  reduce.MergeManagerImpl (MergeManagerImpl.java:finalMerge(795)) - Merging 0 segments, 0 bytes from memory into reduce
2015-02-12 16:12:54,577 INFO  mapred.Merger (Merger.java:merge(606)) - Merging 1 sorted segments
2015-02-12 16:12:54,578 INFO  mapred.Merger (Merger.java:merge(705)) - Down to the last merge-pass, with 1 segments left of total size: 163 bytes
2015-02-12 16:12:54,578 INFO  mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) - 2 / 2 copied.
2015-02-12 16:12:54,578 INFO  hadoop.FlowReducer (FlowReducer.java:configure(78)) - cascading version: 2.6.3
2015-02-12 16:12:54,578 INFO  hadoop.FlowReducer (FlowReducer.java:configure(79)) - child jvm opts: -Xmx200m
2015-02-12 16:12:54,581 INFO  hadoop.FlowReducer (FlowReducer.java:configure(97)) - sourcing from: GroupBy(lhs+rhs)[by: lhs:[{1}:'num'] rhs:[{1}:'num']]
2015-02-12 16:12:54,582 INFO  hadoop.FlowReducer (FlowReducer.java:configure(100)) - sinking to: Hfs["TextLine[['item']->['num', 'char']]"]["/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/test/output/hadoop2-mr1/assemblyhelpersplatform/uniquemerge-nondeterministic"]
2015-02-12 16:12:54,590 INFO  mapred.Task (Task.java:done(1001)) - Task:attempt_local1683387041_0037_r_000000_0 is done. And is in the process of committing
2015-02-12 16:12:54,591 INFO  mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) - 2 / 2 copied.
2015-02-12 16:12:54,591 INFO  mapred.Task (Task.java:commit(1162)) - Task attempt_local1683387041_0037_r_000000_0 is allowed to commit now
2015-02-12 16:12:54,592 INFO  output.FileOutputCommitter (FileOutputCommitter.java:commitTask(439)) - Saved output of task 'attempt_local1683387041_0037_r_000000_0' to file:/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/test/output/hadoop2-mr1/assemblyhelpersplatform/uniquemerge-nondeterministic/_temporary/0/task_local1683387041_0037_r_000000
2015-02-12 16:12:54,593 INFO  mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) - reduce > reduce
2015-02-12 16:12:54,593 INFO  mapred.Task (Task.java:sendDone(1121)) - Task 'attempt_local1683387041_0037_r_000000_0' done.
2015-02-12 16:12:54,593 INFO  mapred.LocalJobRunner (LocalJobRunner.java:run(325)) - Finishing task: attempt_local1683387041_0037_r_000000_0
2015-02-12 16:12:54,593 INFO  mapred.LocalJobRunner (LocalJobRunner.java:runTasks(456)) - reduce task executor complete.
2015-02-12 16:12:54,616 INFO  util.Hadoop18TapUtil (Hadoop18TapUtil.java:cleanTempPath(219)) - deleting temp path /Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/test/output/hadoop2-mr1/assemblyhelpersplatform/uniquemerge-nondeterministic/_temporary
2015-02-12 16:12:54,636 INFO  mapred.FileInputFormat (FileInputFormat.java:listStatus(247)) - Total input paths to process : 1
2015-02-12 16:12:54,637 INFO  cascading.PlatformTestCase (PlatformTestCase.java:tearDown(209)) - copying to local /Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/test/output//hadoop2-mr1/assemblyhelpersplatform/uniquemerge-nondeterministic
2015-02-12 16:12:54,647 INFO  util.HadoopUtil (HadoopUtil.java:findMainClass(336)) - using default application jar, may cause class not found exceptions on the cluster
2015-02-12 16:12:54,647 INFO  planner.HadoopPlanner (HadoopPlanner.java:initialize(225)) - using application jar: /Users/soberoi/.gradle/caches/modules-2/files-2.1/cascading/cascading-hadoop2-mr1/2.6.3/9ae016e2b8d246b9c1de9168e715d9f1b971aa4a/cascading-hadoop2-mr1-2.6.3.jar
2015-02-12 16:12:54,651 INFO  Configuration.deprecation (Configuration.java:warnOnceIfDeprecated(1022)) - mapred.used.genericoptionsparser is deprecated. Instead, use mapreduce.client.genericoptionsparser.used
2015-02-12 16:12:54,652 INFO  Configuration.deprecation (Configuration.java:warnOnceIfDeprecated(1022)) - mapred.job.tracker is deprecated. Instead, use mapreduce.jobtracker.address
2015-02-12 16:12:54,652 INFO  Configuration.deprecation (Configuration.java:warnOnceIfDeprecated(1022)) - mapred.output.compress is deprecated. Instead, use mapreduce.output.fileoutputformat.compress
2015-02-12 16:12:54,655 INFO  flow.Flow (BaseFlow.java:logInfo(1378)) - [shape] starting
2015-02-12 16:12:54,655 INFO  flow.Flow (BaseFlow.java:logInfo(1378)) - [shape]  source: Hfs["TextLine[['offset', 'line']->[ALL]]"]["/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/classes/test/data/lhs.txt"]
2015-02-12 16:12:54,656 INFO  flow.Flow (BaseFlow.java:logInfo(1378)) - [shape]  sink: Hfs["TextLine[['item']->['num', 'char']]"]["/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/test/output/hadoop2-mr1/assemblyhelpersplatform/unique"]
2015-02-12 16:12:54,656 INFO  flow.Flow (BaseFlow.java:logInfo(1378)) - [shape]  parallel execution is enabled: false
2015-02-12 16:12:54,656 INFO  flow.Flow (BaseFlow.java:logInfo(1378)) - [shape]  starting jobs: 1
2015-02-12 16:12:54,656 INFO  flow.Flow (BaseFlow.java:logInfo(1378)) - [shape]  allocating threads: 1
2015-02-12 16:12:54,656 INFO  flow.FlowStep (BaseFlowStep.java:logInfo(834)) - [shape] starting step: (1/1) ...blyhelpersplatform/unique
2015-02-12 16:12:54,657 INFO  jvm.JvmMetrics (JvmMetrics.java:init(71)) - Cannot initialize JVM Metrics with processName=JobTracker, sessionId= - already initialized
2015-02-12 16:12:54,657 INFO  jvm.JvmMetrics (JvmMetrics.java:init(71)) - Cannot initialize JVM Metrics with processName=JobTracker, sessionId= - already initialized
2015-02-12 16:12:54,678 INFO  mapred.FileInputFormat (FileInputFormat.java:listStatus(247)) - Total input paths to process : 1
2015-02-12 16:12:54,693 INFO  mapreduce.JobSubmitter (JobSubmitter.java:submitJobInternal(492)) - number of splits:1
2015-02-12 16:12:54,704 INFO  mapreduce.JobSubmitter (JobSubmitter.java:printTokens(575)) - Submitting tokens for job: job_local2117717862_0038
2015-02-12 16:12:54,713 WARN  conf.Configuration (Configuration.java:loadProperty(2510)) - file:/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/tmp/cascading/staging/soberoi2117717862/.staging/job_local2117717862_0038/job.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.retry.interval;  Ignoring.
2015-02-12 16:12:54,714 WARN  conf.Configuration (Configuration.java:loadProperty(2510)) - file:/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/tmp/cascading/staging/soberoi2117717862/.staging/job_local2117717862_0038/job.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.attempts;  Ignoring.
2015-02-12 16:12:54,741 WARN  conf.Configuration (Configuration.java:loadProperty(2510)) - file:/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/test/tmp/mapred/local/localRunner/soberoi/job_local2117717862_0038/job_local2117717862_0038.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.retry.interval;  Ignoring.
2015-02-12 16:12:54,742 WARN  conf.Configuration (Configuration.java:loadProperty(2510)) - file:/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/test/tmp/mapred/local/localRunner/soberoi/job_local2117717862_0038/job_local2117717862_0038.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.attempts;  Ignoring.
2015-02-12 16:12:54,742 INFO  mapreduce.Job (Job.java:submit(1299)) - The url to track the job: http://localhost:8080/
2015-02-12 16:12:54,743 INFO  mapred.LocalJobRunner (LocalJobRunner.java:createOutputCommitter(471)) - OutputCommitter set in config null
2015-02-12 16:12:54,743 INFO  flow.FlowStep (BaseFlowStep.java:logInfo(834)) - [shape] submitted hadoop job: job_local2117717862_0038
2015-02-12 16:12:54,743 INFO  mapred.LocalJobRunner (LocalJobRunner.java:createOutputCommitter(489)) - OutputCommitter is org.apache.hadoop.mapred.FileOutputCommitter
2015-02-12 16:12:54,743 INFO  flow.FlowStep (BaseFlowStep.java:logInfo(834)) - [shape] tracking url: http://localhost:8080/
2015-02-12 16:12:54,744 INFO  mapred.LocalJobRunner (LocalJobRunner.java:runTasks(448)) - Waiting for map tasks
2015-02-12 16:12:54,744 INFO  mapred.LocalJobRunner (LocalJobRunner.java:run(224)) - Starting task: attempt_local2117717862_0038_m_000000_0
2015-02-12 16:12:54,745 INFO  util.ProcfsBasedProcessTree (ProcfsBasedProcessTree.java:isAvailable(181)) - ProcfsBasedProcessTree currently is supported only on Linux.
2015-02-12 16:12:54,745 INFO  mapred.Task (Task.java:initialize(587)) -  Using ResourceCalculatorProcessTree : null
2015-02-12 16:12:54,746 INFO  io.MultiInputSplit (MultiInputSplit.java:readFields(161)) - current split input path: file:/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/classes/test/data/lhs.txt
2015-02-12 16:12:54,746 INFO  mapred.MapTask (MapTask.java:updateJobWithSplit(482)) - Processing split: cascading.tap.hadoop.io.MultiInputSplit@2a0f331b
2015-02-12 16:12:54,747 INFO  mapred.MapTask (MapTask.java:runOldMapper(436)) - numReduceTasks: 1
2015-02-12 16:12:54,763 INFO  mapred.MapTask (MapTask.java:setEquator(1202)) - (EQUATOR) 0 kvi 26214396(104857584)
2015-02-12 16:12:54,763 INFO  mapred.MapTask (MapTask.java:init(995)) - mapreduce.task.io.sort.mb: 100
2015-02-12 16:12:54,763 INFO  mapred.MapTask (MapTask.java:init(996)) - soft limit at 83886080
2015-02-12 16:12:54,763 INFO  mapred.MapTask (MapTask.java:init(997)) - bufstart = 0; bufvoid = 104857600
2015-02-12 16:12:54,763 INFO  mapred.MapTask (MapTask.java:init(998)) - kvstart = 26214396; length = 6553600
2015-02-12 16:12:54,764 INFO  mapred.MapTask (MapTask.java:createSortingCollector(402)) - Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer
2015-02-12 16:12:54,764 INFO  hadoop.FlowMapper (FlowMapper.java:configure(71)) - cascading version: 2.6.3
2015-02-12 16:12:54,764 INFO  hadoop.FlowMapper (FlowMapper.java:configure(72)) - child jvm opts: -Xmx200m
2015-02-12 16:12:54,766 INFO  hadoop.FlowMapper (FlowMapper.java:configure(87)) - sourcing from: Hfs["TextLine[['offset', 'line']->[ALL]]"]["/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/classes/test/data/lhs.txt"]
2015-02-12 16:12:54,766 INFO  hadoop.FlowMapper (FlowMapper.java:configure(90)) - sinking to: GroupBy(shape)[by:[{1}:'num']]
2015-02-12 16:12:54,768 INFO  mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) - 
2015-02-12 16:12:54,768 INFO  mapred.MapTask (MapTask.java:flush(1457)) - Starting flush of map output
2015-02-12 16:12:54,768 INFO  mapred.MapTask (MapTask.java:flush(1475)) - Spilling map output
2015-02-12 16:12:54,768 INFO  mapred.MapTask (MapTask.java:flush(1476)) - bufstart = 0; bufend = 75; bufvoid = 104857600
2015-02-12 16:12:54,768 INFO  mapred.MapTask (MapTask.java:flush(1478)) - kvstart = 26214396(104857584); kvend = 26214380(104857520); length = 17/6553600
2015-02-12 16:12:54,770 INFO  mapred.MapTask (MapTask.java:sortAndSpill(1660)) - Finished spill 0
2015-02-12 16:12:54,772 INFO  mapred.Task (Task.java:done(1001)) - Task:attempt_local2117717862_0038_m_000000_0 is done. And is in the process of committing
2015-02-12 16:12:54,773 INFO  mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) - file:/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/classes/test/data/lhs.txt:0+52
2015-02-12 16:12:54,773 INFO  mapred.Task (Task.java:sendDone(1121)) - Task 'attempt_local2117717862_0038_m_000000_0' done.
2015-02-12 16:12:54,773 INFO  mapred.LocalJobRunner (LocalJobRunner.java:run(249)) - Finishing task: attempt_local2117717862_0038_m_000000_0
2015-02-12 16:12:54,774 INFO  mapred.LocalJobRunner (LocalJobRunner.java:runTasks(456)) - map task executor complete.
2015-02-12 16:12:54,774 INFO  mapred.LocalJobRunner (LocalJobRunner.java:runTasks(448)) - Waiting for reduce tasks
2015-02-12 16:12:54,774 INFO  mapred.LocalJobRunner (LocalJobRunner.java:run(302)) - Starting task: attempt_local2117717862_0038_r_000000_0
2015-02-12 16:12:54,775 INFO  util.ProcfsBasedProcessTree (ProcfsBasedProcessTree.java:isAvailable(181)) - ProcfsBasedProcessTree currently is supported only on Linux.
2015-02-12 16:12:54,775 INFO  mapred.Task (Task.java:initialize(587)) -  Using ResourceCalculatorProcessTree : null
2015-02-12 16:12:54,775 INFO  mapred.ReduceTask (ReduceTask.java:run(362)) - Using ShuffleConsumerPlugin: org.apache.hadoop.mapreduce.task.reduce.Shuffle@1315391f
2015-02-12 16:12:54,776 INFO  reduce.MergeManagerImpl (MergeManagerImpl.java:<init>(195)) - MergerManager: memoryLimit=493250144, maxSingleShuffleLimit=123312536, mergeThreshold=325545120, ioSortFactor=10, memToMemMergeOutputsThreshold=10
2015-02-12 16:12:54,777 INFO  reduce.EventFetcher (EventFetcher.java:run(61)) - attempt_local2117717862_0038_r_000000_0 Thread started: EventFetcher for fetching Map Completion Events
2015-02-12 16:12:54,777 INFO  reduce.LocalFetcher (LocalFetcher.java:copyMapOutput(141)) - localfetcher#31 about to shuffle output of map attempt_local2117717862_0038_m_000000_0 decomp: 87 len: 91 to MEMORY
2015-02-12 16:12:54,778 INFO  reduce.InMemoryMapOutput (InMemoryMapOutput.java:shuffle(100)) - Read 87 bytes from map-output for attempt_local2117717862_0038_m_000000_0
2015-02-12 16:12:54,778 INFO  reduce.MergeManagerImpl (MergeManagerImpl.java:closeInMemoryFile(313)) - closeInMemoryFile -> map-output of size: 87, inMemoryMapOutputs.size() -> 1, commitMemory -> 0, usedMemory ->87
2015-02-12 16:12:54,778 INFO  reduce.EventFetcher (EventFetcher.java:run(76)) - EventFetcher is interrupted.. Returning
2015-02-12 16:12:54,779 INFO  mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) - 1 / 1 copied.
2015-02-12 16:12:54,779 INFO  reduce.MergeManagerImpl (MergeManagerImpl.java:finalMerge(673)) - finalMerge called with 1 in-memory map-outputs and 0 on-disk map-outputs
2015-02-12 16:12:54,780 INFO  mapred.Merger (Merger.java:merge(606)) - Merging 1 sorted segments
2015-02-12 16:12:54,780 INFO  mapred.Merger (Merger.java:merge(705)) - Down to the last merge-pass, with 1 segments left of total size: 78 bytes
2015-02-12 16:12:54,781 INFO  reduce.MergeManagerImpl (MergeManagerImpl.java:finalMerge(750)) - Merged 1 segments, 87 bytes to disk to satisfy reduce memory limit
2015-02-12 16:12:54,781 INFO  reduce.MergeManagerImpl (MergeManagerImpl.java:finalMerge(780)) - Merging 1 files, 91 bytes from disk
2015-02-12 16:12:54,782 INFO  reduce.MergeManagerImpl (MergeManagerImpl.java:finalMerge(795)) - Merging 0 segments, 0 bytes from memory into reduce
2015-02-12 16:12:54,782 INFO  mapred.Merger (Merger.java:merge(606)) - Merging 1 sorted segments
2015-02-12 16:12:54,782 INFO  mapred.Merger (Merger.java:merge(705)) - Down to the last merge-pass, with 1 segments left of total size: 78 bytes
2015-02-12 16:12:54,782 INFO  mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) - 1 / 1 copied.
2015-02-12 16:12:54,782 INFO  hadoop.FlowReducer (FlowReducer.java:configure(78)) - cascading version: 2.6.3
2015-02-12 16:12:54,783 INFO  hadoop.FlowReducer (FlowReducer.java:configure(79)) - child jvm opts: -Xmx200m
2015-02-12 16:12:54,785 INFO  hadoop.FlowReducer (FlowReducer.java:configure(97)) - sourcing from: GroupBy(shape)[by:[{1}:'num']]
2015-02-12 16:12:54,785 INFO  hadoop.FlowReducer (FlowReducer.java:configure(100)) - sinking to: Hfs["TextLine[['item']->['num', 'char']]"]["/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/test/output/hadoop2-mr1/assemblyhelpersplatform/unique"]
2015-02-12 16:12:54,791 INFO  mapred.Task (Task.java:done(1001)) - Task:attempt_local2117717862_0038_r_000000_0 is done. And is in the process of committing
2015-02-12 16:12:54,792 INFO  mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) - 1 / 1 copied.
2015-02-12 16:12:54,792 INFO  mapred.Task (Task.java:commit(1162)) - Task attempt_local2117717862_0038_r_000000_0 is allowed to commit now
2015-02-12 16:12:54,793 INFO  output.FileOutputCommitter (FileOutputCommitter.java:commitTask(439)) - Saved output of task 'attempt_local2117717862_0038_r_000000_0' to file:/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/test/output/hadoop2-mr1/assemblyhelpersplatform/unique/_temporary/0/task_local2117717862_0038_r_000000
2015-02-12 16:12:54,793 INFO  mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) - reduce > reduce
2015-02-12 16:12:54,793 INFO  mapred.Task (Task.java:sendDone(1121)) - Task 'attempt_local2117717862_0038_r_000000_0' done.
2015-02-12 16:12:54,794 INFO  mapred.LocalJobRunner (LocalJobRunner.java:run(325)) - Finishing task: attempt_local2117717862_0038_r_000000_0
2015-02-12 16:12:54,794 INFO  mapred.LocalJobRunner (LocalJobRunner.java:runTasks(456)) - reduce task executor complete.
2015-02-12 16:12:54,804 INFO  util.Hadoop18TapUtil (Hadoop18TapUtil.java:cleanTempPath(219)) - deleting temp path /Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/test/output/hadoop2-mr1/assemblyhelpersplatform/unique/_temporary
2015-02-12 16:12:54,824 INFO  mapred.FileInputFormat (FileInputFormat.java:listStatus(247)) - Total input paths to process : 1
2015-02-12 16:12:54,825 INFO  cascading.PlatformTestCase (PlatformTestCase.java:tearDown(209)) - copying to local /Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/test/output//hadoop2-mr1/assemblyhelpersplatform/unique
2015-02-12 16:12:54,837 INFO  util.HadoopUtil (HadoopUtil.java:findMainClass(336)) - using default application jar, may cause class not found exceptions on the cluster
2015-02-12 16:12:54,837 INFO  planner.HadoopPlanner (HadoopPlanner.java:initialize(225)) - using application jar: /Users/soberoi/.gradle/caches/modules-2/files-2.1/cascading/cascading-hadoop2-mr1/2.6.3/9ae016e2b8d246b9c1de9168e715d9f1b971aa4a/cascading-hadoop2-mr1-2.6.3.jar
2015-02-12 16:12:54,840 INFO  Configuration.deprecation (Configuration.java:warnOnceIfDeprecated(1022)) - mapred.used.genericoptionsparser is deprecated. Instead, use mapreduce.client.genericoptionsparser.used
2015-02-12 16:12:54,841 INFO  Configuration.deprecation (Configuration.java:warnOnceIfDeprecated(1022)) - mapred.job.tracker is deprecated. Instead, use mapreduce.jobtracker.address
2015-02-12 16:12:54,841 INFO  Configuration.deprecation (Configuration.java:warnOnceIfDeprecated(1022)) - mapred.output.compress is deprecated. Instead, use mapreduce.output.fileoutputformat.compress
2015-02-12 16:12:54,845 INFO  flow.Flow (BaseFlow.java:logInfo(1378)) - [count] starting
2015-02-12 16:12:54,845 INFO  flow.Flow (BaseFlow.java:logInfo(1378)) - [count]  source: Hfs["TextDelimited[['num', 'char']]"]["/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/classes/test/data/lhs.txt"]
2015-02-12 16:12:54,845 INFO  flow.Flow (BaseFlow.java:logInfo(1378)) - [count]  sink: Hfs["TextDelimited[['notnull', 'null']]"]["/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/test/output/hadoop2-mr1/assemblyhelpersplatform/countnullnotnull"]
2015-02-12 16:12:54,845 INFO  flow.Flow (BaseFlow.java:logInfo(1378)) - [count]  parallel execution is enabled: false
2015-02-12 16:12:54,845 INFO  flow.Flow (BaseFlow.java:logInfo(1378)) - [count]  starting jobs: 1
2015-02-12 16:12:54,845 INFO  flow.Flow (BaseFlow.java:logInfo(1378)) - [count]  allocating threads: 1
2015-02-12 16:12:54,846 INFO  flow.FlowStep (BaseFlowStep.java:logInfo(834)) - [count] starting step: (1/1) ...platform/countnullnotnull
2015-02-12 16:12:54,846 INFO  jvm.JvmMetrics (JvmMetrics.java:init(71)) - Cannot initialize JVM Metrics with processName=JobTracker, sessionId= - already initialized
2015-02-12 16:12:54,847 INFO  jvm.JvmMetrics (JvmMetrics.java:init(71)) - Cannot initialize JVM Metrics with processName=JobTracker, sessionId= - already initialized
2015-02-12 16:12:54,865 INFO  mapred.FileInputFormat (FileInputFormat.java:listStatus(247)) - Total input paths to process : 1
2015-02-12 16:12:54,881 INFO  mapreduce.JobSubmitter (JobSubmitter.java:submitJobInternal(492)) - number of splits:1
2015-02-12 16:12:54,892 INFO  mapreduce.JobSubmitter (JobSubmitter.java:printTokens(575)) - Submitting tokens for job: job_local1121988028_0039
2015-02-12 16:12:54,901 WARN  conf.Configuration (Configuration.java:loadProperty(2510)) - file:/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/tmp/cascading/staging/soberoi1121988028/.staging/job_local1121988028_0039/job.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.retry.interval;  Ignoring.
2015-02-12 16:12:54,903 WARN  conf.Configuration (Configuration.java:loadProperty(2510)) - file:/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/tmp/cascading/staging/soberoi1121988028/.staging/job_local1121988028_0039/job.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.attempts;  Ignoring.
2015-02-12 16:12:54,934 WARN  conf.Configuration (Configuration.java:loadProperty(2510)) - file:/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/test/tmp/mapred/local/localRunner/soberoi/job_local1121988028_0039/job_local1121988028_0039.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.retry.interval;  Ignoring.
2015-02-12 16:12:54,935 WARN  conf.Configuration (Configuration.java:loadProperty(2510)) - file:/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/test/tmp/mapred/local/localRunner/soberoi/job_local1121988028_0039/job_local1121988028_0039.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.attempts;  Ignoring.
2015-02-12 16:12:54,935 INFO  mapreduce.Job (Job.java:submit(1299)) - The url to track the job: http://localhost:8080/
2015-02-12 16:12:54,935 INFO  flow.FlowStep (BaseFlowStep.java:logInfo(834)) - [count] submitted hadoop job: job_local1121988028_0039
2015-02-12 16:12:54,935 INFO  mapred.LocalJobRunner (LocalJobRunner.java:createOutputCommitter(471)) - OutputCommitter set in config null
2015-02-12 16:12:54,935 INFO  flow.FlowStep (BaseFlowStep.java:logInfo(834)) - [count] tracking url: http://localhost:8080/
2015-02-12 16:12:54,936 INFO  mapred.LocalJobRunner (LocalJobRunner.java:createOutputCommitter(489)) - OutputCommitter is org.apache.hadoop.mapred.FileOutputCommitter
2015-02-12 16:12:54,937 INFO  mapred.LocalJobRunner (LocalJobRunner.java:runTasks(448)) - Waiting for map tasks
2015-02-12 16:12:54,937 INFO  mapred.LocalJobRunner (LocalJobRunner.java:run(224)) - Starting task: attempt_local1121988028_0039_m_000000_0
2015-02-12 16:12:54,938 INFO  util.ProcfsBasedProcessTree (ProcfsBasedProcessTree.java:isAvailable(181)) - ProcfsBasedProcessTree currently is supported only on Linux.
2015-02-12 16:12:54,938 INFO  mapred.Task (Task.java:initialize(587)) -  Using ResourceCalculatorProcessTree : null
2015-02-12 16:12:54,939 INFO  io.MultiInputSplit (MultiInputSplit.java:readFields(161)) - current split input path: file:/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/classes/test/data/lhs.txt
2015-02-12 16:12:54,939 INFO  mapred.MapTask (MapTask.java:updateJobWithSplit(482)) - Processing split: cascading.tap.hadoop.io.MultiInputSplit@63b30992
2015-02-12 16:12:54,939 INFO  mapred.MapTask (MapTask.java:runOldMapper(436)) - numReduceTasks: 1
2015-02-12 16:12:54,955 INFO  mapred.MapTask (MapTask.java:setEquator(1202)) - (EQUATOR) 0 kvi 26214396(104857584)
2015-02-12 16:12:54,955 INFO  mapred.MapTask (MapTask.java:init(995)) - mapreduce.task.io.sort.mb: 100
2015-02-12 16:12:54,955 INFO  mapred.MapTask (MapTask.java:init(996)) - soft limit at 83886080
2015-02-12 16:12:54,955 INFO  mapred.MapTask (MapTask.java:init(997)) - bufstart = 0; bufvoid = 104857600
2015-02-12 16:12:54,955 INFO  mapred.MapTask (MapTask.java:init(998)) - kvstart = 26214396; length = 6553600
2015-02-12 16:12:54,955 INFO  mapred.MapTask (MapTask.java:createSortingCollector(402)) - Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer
2015-02-12 16:12:54,956 INFO  hadoop.FlowMapper (FlowMapper.java:configure(71)) - cascading version: 2.6.3
2015-02-12 16:12:54,956 INFO  hadoop.FlowMapper (FlowMapper.java:configure(72)) - child jvm opts: -Xmx200m
2015-02-12 16:12:54,959 INFO  hadoop.FlowMapper (FlowMapper.java:configure(87)) - sourcing from: Hfs["TextDelimited[['num', 'char']]"]["/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/classes/test/data/lhs.txt"]
2015-02-12 16:12:54,959 INFO  hadoop.FlowMapper (FlowMapper.java:configure(90)) - sinking to: GroupBy(count)[by:[{?}:NONE]]
2015-02-12 16:12:54,959 INFO  assembly.AggregateBy (AggregateBy.java:prepare(267)) - using threshold value: 2
2015-02-12 16:12:54,962 INFO  mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) - 
2015-02-12 16:12:54,962 INFO  mapred.MapTask (MapTask.java:flush(1457)) - Starting flush of map output
2015-02-12 16:12:54,962 INFO  mapred.MapTask (MapTask.java:flush(1475)) - Spilling map output
2015-02-12 16:12:54,963 INFO  mapred.MapTask (MapTask.java:flush(1476)) - bufstart = 0; bufend = 6; bufvoid = 104857600
2015-02-12 16:12:54,963 INFO  mapred.MapTask (MapTask.java:flush(1478)) - kvstart = 26214396(104857584); kvend = 26214396(104857584); length = 1/6553600
2015-02-12 16:12:54,964 INFO  mapred.MapTask (MapTask.java:sortAndSpill(1660)) - Finished spill 0
2015-02-12 16:12:54,965 INFO  mapred.Task (Task.java:done(1001)) - Task:attempt_local1121988028_0039_m_000000_0 is done. And is in the process of committing
2015-02-12 16:12:54,966 INFO  mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) - file:/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/classes/test/data/lhs.txt:0+52
2015-02-12 16:12:54,966 INFO  mapred.Task (Task.java:sendDone(1121)) - Task 'attempt_local1121988028_0039_m_000000_0' done.
2015-02-12 16:12:54,966 INFO  mapred.LocalJobRunner (LocalJobRunner.java:run(249)) - Finishing task: attempt_local1121988028_0039_m_000000_0
2015-02-12 16:12:54,966 INFO  mapred.LocalJobRunner (LocalJobRunner.java:runTasks(456)) - map task executor complete.
2015-02-12 16:12:54,967 INFO  mapred.LocalJobRunner (LocalJobRunner.java:runTasks(448)) - Waiting for reduce tasks
2015-02-12 16:12:54,967 INFO  mapred.LocalJobRunner (LocalJobRunner.java:run(302)) - Starting task: attempt_local1121988028_0039_r_000000_0
2015-02-12 16:12:54,968 INFO  util.ProcfsBasedProcessTree (ProcfsBasedProcessTree.java:isAvailable(181)) - ProcfsBasedProcessTree currently is supported only on Linux.
2015-02-12 16:12:54,968 INFO  mapred.Task (Task.java:initialize(587)) -  Using ResourceCalculatorProcessTree : null
2015-02-12 16:12:54,968 INFO  mapred.ReduceTask (ReduceTask.java:run(362)) - Using ShuffleConsumerPlugin: org.apache.hadoop.mapreduce.task.reduce.Shuffle@6db52762
2015-02-12 16:12:54,968 INFO  reduce.MergeManagerImpl (MergeManagerImpl.java:<init>(195)) - MergerManager: memoryLimit=493250144, maxSingleShuffleLimit=123312536, mergeThreshold=325545120, ioSortFactor=10, memToMemMergeOutputsThreshold=10
2015-02-12 16:12:54,969 INFO  reduce.EventFetcher (EventFetcher.java:run(61)) - attempt_local1121988028_0039_r_000000_0 Thread started: EventFetcher for fetching Map Completion Events
2015-02-12 16:12:54,969 INFO  reduce.LocalFetcher (LocalFetcher.java:copyMapOutput(141)) - localfetcher#32 about to shuffle output of map attempt_local1121988028_0039_m_000000_0 decomp: 10 len: 14 to MEMORY
2015-02-12 16:12:54,970 INFO  reduce.InMemoryMapOutput (InMemoryMapOutput.java:shuffle(100)) - Read 10 bytes from map-output for attempt_local1121988028_0039_m_000000_0
2015-02-12 16:12:54,970 INFO  reduce.MergeManagerImpl (MergeManagerImpl.java:closeInMemoryFile(313)) - closeInMemoryFile -> map-output of size: 10, inMemoryMapOutputs.size() -> 1, commitMemory -> 0, usedMemory ->10
2015-02-12 16:12:54,970 INFO  reduce.EventFetcher (EventFetcher.java:run(76)) - EventFetcher is interrupted.. Returning
2015-02-12 16:12:54,971 INFO  mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) - 1 / 1 copied.
2015-02-12 16:12:54,971 INFO  reduce.MergeManagerImpl (MergeManagerImpl.java:finalMerge(673)) - finalMerge called with 1 in-memory map-outputs and 0 on-disk map-outputs
2015-02-12 16:12:54,972 INFO  mapred.Merger (Merger.java:merge(606)) - Merging 1 sorted segments
2015-02-12 16:12:54,972 INFO  mapred.Merger (Merger.java:merge(705)) - Down to the last merge-pass, with 1 segments left of total size: 7 bytes
2015-02-12 16:12:54,973 INFO  reduce.MergeManagerImpl (MergeManagerImpl.java:finalMerge(750)) - Merged 1 segments, 10 bytes to disk to satisfy reduce memory limit
2015-02-12 16:12:54,973 INFO  reduce.MergeManagerImpl (MergeManagerImpl.java:finalMerge(780)) - Merging 1 files, 14 bytes from disk
2015-02-12 16:12:54,973 INFO  reduce.MergeManagerImpl (MergeManagerImpl.java:finalMerge(795)) - Merging 0 segments, 0 bytes from memory into reduce
2015-02-12 16:12:54,973 INFO  mapred.Merger (Merger.java:merge(606)) - Merging 1 sorted segments
2015-02-12 16:12:54,973 INFO  mapred.Merger (Merger.java:merge(705)) - Down to the last merge-pass, with 1 segments left of total size: 7 bytes
2015-02-12 16:12:54,974 INFO  mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) - 1 / 1 copied.
2015-02-12 16:12:54,974 INFO  hadoop.FlowReducer (FlowReducer.java:configure(78)) - cascading version: 2.6.3
2015-02-12 16:12:54,974 INFO  hadoop.FlowReducer (FlowReducer.java:configure(79)) - child jvm opts: -Xmx200m
2015-02-12 16:12:54,978 INFO  hadoop.FlowReducer (FlowReducer.java:configure(97)) - sourcing from: GroupBy(count)[by:[{?}:NONE]]
2015-02-12 16:12:54,978 INFO  hadoop.FlowReducer (FlowReducer.java:configure(100)) - sinking to: Hfs["TextDelimited[['notnull', 'null']]"]["/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/test/output/hadoop2-mr1/assemblyhelpersplatform/countnullnotnull"]
2015-02-12 16:12:54,984 INFO  mapred.Task (Task.java:done(1001)) - Task:attempt_local1121988028_0039_r_000000_0 is done. And is in the process of committing
2015-02-12 16:12:54,985 INFO  mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) - 1 / 1 copied.
2015-02-12 16:12:54,986 INFO  mapred.Task (Task.java:commit(1162)) - Task attempt_local1121988028_0039_r_000000_0 is allowed to commit now
2015-02-12 16:12:54,986 INFO  output.FileOutputCommitter (FileOutputCommitter.java:commitTask(439)) - Saved output of task 'attempt_local1121988028_0039_r_000000_0' to file:/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/test/output/hadoop2-mr1/assemblyhelpersplatform/countnullnotnull/_temporary/0/task_local1121988028_0039_r_000000
2015-02-12 16:12:54,987 INFO  mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) - reduce > reduce
2015-02-12 16:12:54,987 INFO  mapred.Task (Task.java:sendDone(1121)) - Task 'attempt_local1121988028_0039_r_000000_0' done.
2015-02-12 16:12:54,987 INFO  mapred.LocalJobRunner (LocalJobRunner.java:run(325)) - Finishing task: attempt_local1121988028_0039_r_000000_0
2015-02-12 16:12:54,987 INFO  mapred.LocalJobRunner (LocalJobRunner.java:runTasks(456)) - reduce task executor complete.
2015-02-12 16:12:55,006 INFO  util.Hadoop18TapUtil (Hadoop18TapUtil.java:cleanTempPath(219)) - deleting temp path /Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/test/output/hadoop2-mr1/assemblyhelpersplatform/countnullnotnull/_temporary
2015-02-12 16:12:55,026 INFO  mapred.FileInputFormat (FileInputFormat.java:listStatus(247)) - Total input paths to process : 1
2015-02-12 16:12:55,046 INFO  mapred.FileInputFormat (FileInputFormat.java:listStatus(247)) - Total input paths to process : 1
2015-02-12 16:12:55,047 INFO  cascading.PlatformTestCase (PlatformTestCase.java:tearDown(209)) - copying to local /Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/test/output//hadoop2-mr1/assemblyhelpersplatform/countnullnotnull