2015-02-12 16:12:43,740 INFO platform.PlatformRunner (PlatformRunner.java:getPlatformClassesFromClasspath(187)) - found 1 test platforms from classpath
2015-02-12 16:12:43,773 INFO platform.PlatformRunner (PlatformRunner.java:addPlatform(237)) - installing platform: hadoop2-mr1
2015-02-12 16:12:43,773 INFO platform.PlatformRunner (PlatformRunner.java:addPlatform(238)) - running test: cascading.pipe.assembly.AssemblyHelpersPlatformTest
2015-02-12 16:12:43,795 INFO hadoop2.Hadoop2MR1Platform (Hadoop2MR1Platform.java:setUp(105)) - not using cluster
2015-02-12 16:12:44,230 WARN util.NativeCodeLoader (NativeCodeLoader.java:<clinit>(62)) - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
2015-02-12 16:12:44,570 INFO util.HadoopUtil (HadoopUtil.java:findMainClass(336)) - using default application jar, may cause class not found exceptions on the cluster
2015-02-12 16:12:44,571 INFO planner.HadoopPlanner (HadoopPlanner.java:initialize(225)) - using application jar: /Users/soberoi/.gradle/caches/modules-2/files-2.1/cascading/cascading-hadoop2-mr1/2.6.3/9ae016e2b8d246b9c1de9168e715d9f1b971aa4a/cascading-hadoop2-mr1-2.6.3.jar
2015-02-12 16:12:44,578 INFO property.AppProps (AppProps.java:getAppID(169)) - using app.id: AE80D2D9424D4C76941AE218CDDD15C7
2015-02-12 16:12:44,662 INFO Configuration.deprecation (Configuration.java:warnOnceIfDeprecated(1022)) - mapred.used.genericoptionsparser is deprecated. Instead, use mapreduce.client.genericoptionsparser.used
2015-02-12 16:12:44,692 INFO Configuration.deprecation (Configuration.java:warnOnceIfDeprecated(1022)) - mapred.job.tracker is deprecated. Instead, use mapreduce.jobtracker.address
2015-02-12 16:12:44,695 INFO Configuration.deprecation (Configuration.java:warnOnceIfDeprecated(1022)) - mapred.output.compress is deprecated. Instead, use mapreduce.output.fileoutputformat.compress
2015-02-12 16:12:44,804 INFO util.Version (Version.java:printBanner(82)) - Concurrent, Inc - Cascading 2.6.3
2015-02-12 16:12:44,806 INFO flow.Flow (BaseFlow.java:logInfo(1378)) - [name] starting
2015-02-12 16:12:44,806 INFO flow.Flow (BaseFlow.java:logInfo(1378)) - [name] source: Hfs["TextDelimited[['num', 'char']]"]["/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/classes/test/data/rhs.txt"]
2015-02-12 16:12:44,807 INFO flow.Flow (BaseFlow.java:logInfo(1378)) - [name] source: Hfs["TextDelimited[['num', 'char']]"]["/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/classes/test/data/lhs.txt"]
2015-02-12 16:12:44,807 INFO flow.Flow (BaseFlow.java:logInfo(1378)) - [name] sink: Hfs["TextDelimited[['char', 'sum', 'count', 'average']]"]["/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/test/output/hadoop2-mr1/assemblyhelpersplatform/multimerge+false"]
2015-02-12 16:12:44,808 INFO flow.Flow (BaseFlow.java:logInfo(1378)) - [name] parallel execution is enabled: false
2015-02-12 16:12:44,808 INFO flow.Flow (BaseFlow.java:logInfo(1378)) - [name] starting jobs: 1
2015-02-12 16:12:44,808 INFO flow.Flow (BaseFlow.java:logInfo(1378)) - [name] allocating threads: 1
2015-02-12 16:12:44,809 INFO flow.FlowStep (BaseFlowStep.java:logInfo(834)) - [name] starting step: (1/1) ...platform/multimerge+false
2015-02-12 16:12:44,830 INFO Configuration.deprecation (Configuration.java:warnOnceIfDeprecated(1022)) - session.id is deprecated. Instead, use dfs.metrics.session-id
2015-02-12 16:12:44,831 INFO jvm.JvmMetrics (JvmMetrics.java:init(76)) - Initializing JVM Metrics with processName=JobTracker, sessionId=
2015-02-12 16:12:44,845 INFO jvm.JvmMetrics (JvmMetrics.java:init(71)) - Cannot initialize JVM Metrics with processName=JobTracker, sessionId= - already initialized
2015-02-12 16:12:45,002 INFO mapred.FileInputFormat (FileInputFormat.java:listStatus(247)) - Total input paths to process : 1
2015-02-12 16:12:45,012 INFO mapred.FileInputFormat (FileInputFormat.java:listStatus(247)) - Total input paths to process : 1
2015-02-12 16:12:45,039 INFO mapreduce.JobSubmitter (JobSubmitter.java:submitJobInternal(492)) - number of splits:2
2015-02-12 16:12:45,220 INFO mapreduce.JobSubmitter (JobSubmitter.java:printTokens(575)) - Submitting tokens for job: job_local1069201491_0001
2015-02-12 16:12:45,275 WARN conf.Configuration (Configuration.java:loadProperty(2510)) - file:/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/tmp/cascading/staging/soberoi1069201491/.staging/job_local1069201491_0001/job.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.retry.interval; Ignoring.
2015-02-12 16:12:45,280 WARN conf.Configuration (Configuration.java:loadProperty(2510)) - file:/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/tmp/cascading/staging/soberoi1069201491/.staging/job_local1069201491_0001/job.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.attempts; Ignoring.
2015-02-12 16:12:45,357 WARN conf.Configuration (Configuration.java:loadProperty(2510)) - file:/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/test/tmp/mapred/local/localRunner/soberoi/job_local1069201491_0001/job_local1069201491_0001.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.retry.interval; Ignoring.
2015-02-12 16:12:45,359 WARN conf.Configuration (Configuration.java:loadProperty(2510)) - file:/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/test/tmp/mapred/local/localRunner/soberoi/job_local1069201491_0001/job_local1069201491_0001.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.attempts; Ignoring.
2015-02-12 16:12:45,362 INFO mapreduce.Job (Job.java:submit(1299)) - The url to track the job: http://localhost:8080/
2015-02-12 16:12:45,363 INFO mapred.LocalJobRunner (LocalJobRunner.java:createOutputCommitter(471)) - OutputCommitter set in config null
2015-02-12 16:12:45,364 INFO flow.FlowStep (BaseFlowStep.java:logInfo(834)) - [name] submitted hadoop job: job_local1069201491_0001
2015-02-12 16:12:45,364 INFO mapred.LocalJobRunner (LocalJobRunner.java:createOutputCommitter(489)) - OutputCommitter is org.apache.hadoop.mapred.FileOutputCommitter
2015-02-12 16:12:45,364 INFO flow.FlowStep (BaseFlowStep.java:logInfo(834)) - [name] tracking url: http://localhost:8080/
2015-02-12 16:12:45,375 INFO mapred.LocalJobRunner (LocalJobRunner.java:runTasks(448)) - Waiting for map tasks
2015-02-12 16:12:45,377 INFO mapred.LocalJobRunner (LocalJobRunner.java:run(224)) - Starting task: attempt_local1069201491_0001_m_000000_0
2015-02-12 16:12:45,394 INFO util.ProcfsBasedProcessTree (ProcfsBasedProcessTree.java:isAvailable(181)) - ProcfsBasedProcessTree currently is supported only on Linux.
2015-02-12 16:12:45,395 INFO mapred.Task (Task.java:initialize(587)) - Using ResourceCalculatorProcessTree : null
2015-02-12 16:12:45,399 INFO io.MultiInputSplit (MultiInputSplit.java:readFields(161)) - current split input path: file:/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/classes/test/data/lhs.txt
2015-02-12 16:12:45,399 INFO mapred.MapTask (MapTask.java:updateJobWithSplit(482)) - Processing split: cascading.tap.hadoop.io.MultiInputSplit@20d77214
2015-02-12 16:12:45,404 INFO mapred.MapTask (MapTask.java:runOldMapper(436)) - numReduceTasks: 1
2015-02-12 16:12:45,438 INFO mapred.MapTask (MapTask.java:setEquator(1202)) - (EQUATOR) 0 kvi 26214396(104857584)
2015-02-12 16:12:45,438 INFO mapred.MapTask (MapTask.java:init(995)) - mapreduce.task.io.sort.mb: 100
2015-02-12 16:12:45,438 INFO mapred.MapTask (MapTask.java:init(996)) - soft limit at 83886080
2015-02-12 16:12:45,438 INFO mapred.MapTask (MapTask.java:init(997)) - bufstart = 0; bufvoid = 104857600
2015-02-12 16:12:45,438 INFO mapred.MapTask (MapTask.java:init(998)) - kvstart = 26214396; length = 6553600
2015-02-12 16:12:45,453 INFO mapred.MapTask (MapTask.java:createSortingCollector(402)) - Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer
2015-02-12 16:12:45,454 INFO hadoop.FlowMapper (FlowMapper.java:configure(71)) - cascading version: 2.6.3
2015-02-12 16:12:45,454 INFO hadoop.FlowMapper (FlowMapper.java:configure(72)) - child jvm opts: -Xmx200m
2015-02-12 16:12:45,484 INFO Configuration.deprecation (Configuration.java:warnOnceIfDeprecated(1022)) - mapred.task.partition is deprecated. Instead, use mapreduce.task.partition
2015-02-12 16:12:45,488 INFO hadoop.FlowMapper (FlowMapper.java:configure(87)) - sourcing from: Hfs["TextDelimited[['num', 'char']]"]["/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/classes/test/data/lhs.txt"]
2015-02-12 16:12:45,488 INFO hadoop.FlowMapper (FlowMapper.java:configure(90)) - sinking to: GroupBy(name)[by: multi-lhs:[{1}:'char'] multi-rhs:[{1}:'char']]
2015-02-12 16:12:45,490 INFO assembly.AggregateBy (AggregateBy.java:prepare(267)) - using threshold value: 2
2015-02-12 16:12:45,498 INFO assembly.AggregateBy (AggregateBy.java:removeEldestEntry(327)) - flushed keys num times: 1, with threshold: 2
2015-02-12 16:12:45,498 INFO assembly.AggregateBy (AggregateBy.java:removeEldestEntry(328)) - mem on flush (mb), free: 172, total: 310, max: 672
2015-02-12 16:12:45,498 INFO assembly.AggregateBy (AggregateBy.java:removeEldestEntry(333)) - total mem is 46% of max mem, to better utilize unused memory consider increasing current LRU threshold with system property "cascading.aggregateby.threshold"
2015-02-12 16:12:45,499 INFO assembly.AggregateBy (AggregateBy.java:removeEldestEntry(327)) - flushed keys num times: 3, with threshold: 2
2015-02-12 16:12:45,499 INFO assembly.AggregateBy (AggregateBy.java:removeEldestEntry(328)) - mem on flush (mb), free: 172, total: 310, max: 672
2015-02-12 16:12:45,499 INFO assembly.AggregateBy (AggregateBy.java:removeEldestEntry(333)) - total mem is 46% of max mem, to better utilize unused memory consider increasing current LRU threshold with system property "cascading.aggregateby.threshold"
2015-02-12 16:12:45,500 INFO assembly.AggregateBy (AggregateBy.java:removeEldestEntry(327)) - flushed keys num times: 5, with threshold: 2
2015-02-12 16:12:45,500 INFO assembly.AggregateBy (AggregateBy.java:removeEldestEntry(328)) - mem on flush (mb), free: 172, total: 310, max: 672
2015-02-12 16:12:45,500 INFO assembly.AggregateBy (AggregateBy.java:removeEldestEntry(333)) - total mem is 46% of max mem, to better utilize unused memory consider increasing current LRU threshold with system property "cascading.aggregateby.threshold"
2015-02-12 16:12:45,501 INFO assembly.AggregateBy (AggregateBy.java:removeEldestEntry(327)) - flushed keys num times: 7, with threshold: 2
2015-02-12 16:12:45,501 INFO assembly.AggregateBy (AggregateBy.java:removeEldestEntry(328)) - mem on flush (mb), free: 172, total: 310, max: 672
2015-02-12 16:12:45,501 INFO assembly.AggregateBy (AggregateBy.java:removeEldestEntry(333)) - total mem is 46% of max mem, to better utilize unused memory consider increasing current LRU threshold with system property "cascading.aggregateby.threshold"
2015-02-12 16:12:45,505 INFO mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) -
2015-02-12 16:12:45,505 INFO mapred.MapTask (MapTask.java:flush(1457)) - Starting flush of map output
2015-02-12 16:12:45,505 INFO mapred.MapTask (MapTask.java:flush(1475)) - Spilling map output
2015-02-12 16:12:45,505 INFO mapred.MapTask (MapTask.java:flush(1476)) - bufstart = 0; bufend = 216; bufvoid = 104857600
2015-02-12 16:12:45,505 INFO mapred.MapTask (MapTask.java:flush(1478)) - kvstart = 26214396(104857584); kvend = 26214364(104857456); length = 33/6553600
2015-02-12 16:12:45,511 INFO mapred.MapTask (MapTask.java:sortAndSpill(1660)) - Finished spill 0
2015-02-12 16:12:45,514 INFO mapred.Task (Task.java:done(1001)) - Task:attempt_local1069201491_0001_m_000000_0 is done. And is in the process of committing
2015-02-12 16:12:45,521 INFO mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) - file:/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/classes/test/data/lhs.txt:0+52
2015-02-12 16:12:45,521 INFO mapred.Task (Task.java:sendDone(1121)) - Task 'attempt_local1069201491_0001_m_000000_0' done.
2015-02-12 16:12:45,521 INFO mapred.LocalJobRunner (LocalJobRunner.java:run(249)) - Finishing task: attempt_local1069201491_0001_m_000000_0
2015-02-12 16:12:45,521 INFO mapred.LocalJobRunner (LocalJobRunner.java:run(224)) - Starting task: attempt_local1069201491_0001_m_000001_0
2015-02-12 16:12:45,524 INFO util.ProcfsBasedProcessTree (ProcfsBasedProcessTree.java:isAvailable(181)) - ProcfsBasedProcessTree currently is supported only on Linux.
2015-02-12 16:12:45,524 INFO mapred.Task (Task.java:initialize(587)) - Using ResourceCalculatorProcessTree : null
2015-02-12 16:12:45,526 INFO io.MultiInputSplit (MultiInputSplit.java:readFields(161)) - current split input path: file:/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/classes/test/data/rhs.txt
2015-02-12 16:12:45,526 INFO mapred.MapTask (MapTask.java:updateJobWithSplit(482)) - Processing split: cascading.tap.hadoop.io.MultiInputSplit@64983afa
2015-02-12 16:12:45,528 INFO mapred.MapTask (MapTask.java:runOldMapper(436)) - numReduceTasks: 1
2015-02-12 16:12:45,571 INFO mapred.MapTask (MapTask.java:setEquator(1202)) - (EQUATOR) 0 kvi 26214396(104857584)
2015-02-12 16:12:45,571 INFO mapred.MapTask (MapTask.java:init(995)) - mapreduce.task.io.sort.mb: 100
2015-02-12 16:12:45,571 INFO mapred.MapTask (MapTask.java:init(996)) - soft limit at 83886080
2015-02-12 16:12:45,571 INFO mapred.MapTask (MapTask.java:init(997)) - bufstart = 0; bufvoid = 104857600
2015-02-12 16:12:45,571 INFO mapred.MapTask (MapTask.java:init(998)) - kvstart = 26214396; length = 6553600
2015-02-12 16:12:45,573 INFO mapred.MapTask (MapTask.java:createSortingCollector(402)) - Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer
2015-02-12 16:12:45,574 INFO hadoop.FlowMapper (FlowMapper.java:configure(71)) - cascading version: 2.6.3
2015-02-12 16:12:45,574 INFO hadoop.FlowMapper (FlowMapper.java:configure(72)) - child jvm opts: -Xmx200m
2015-02-12 16:12:45,588 INFO hadoop.FlowMapper (FlowMapper.java:configure(87)) - sourcing from: Hfs["TextDelimited[['num', 'char']]"]["/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/classes/test/data/rhs.txt"]
2015-02-12 16:12:45,588 INFO hadoop.FlowMapper (FlowMapper.java:configure(90)) - sinking to: GroupBy(name)[by: multi-lhs:[{1}:'char'] multi-rhs:[{1}:'char']]
2015-02-12 16:12:45,588 INFO assembly.AggregateBy (AggregateBy.java:prepare(267)) - using threshold value: 2
2015-02-12 16:12:45,663 INFO assembly.AggregateBy (AggregateBy.java:removeEldestEntry(327)) - flushed keys num times: 1, with threshold: 2
2015-02-12 16:12:45,663 INFO assembly.AggregateBy (AggregateBy.java:removeEldestEntry(328)) - mem on flush (mb), free: 198, total: 310, max: 672
2015-02-12 16:12:45,663 INFO assembly.AggregateBy (AggregateBy.java:removeEldestEntry(333)) - total mem is 46% of max mem, to better utilize unused memory consider increasing current LRU threshold with system property "cascading.aggregateby.threshold"
2015-02-12 16:12:45,664 INFO assembly.AggregateBy (AggregateBy.java:removeEldestEntry(327)) - flushed keys num times: 3, with threshold: 2
2015-02-12 16:12:45,664 INFO assembly.AggregateBy (AggregateBy.java:removeEldestEntry(328)) - mem on flush (mb), free: 197, total: 310, max: 672
2015-02-12 16:12:45,664 INFO assembly.AggregateBy (AggregateBy.java:removeEldestEntry(333)) - total mem is 46% of max mem, to better utilize unused memory consider increasing current LRU threshold with system property "cascading.aggregateby.threshold"
2015-02-12 16:12:45,665 INFO assembly.AggregateBy (AggregateBy.java:removeEldestEntry(327)) - flushed keys num times: 5, with threshold: 2
2015-02-12 16:12:45,665 INFO assembly.AggregateBy (AggregateBy.java:removeEldestEntry(328)) - mem on flush (mb), free: 197, total: 310, max: 672
2015-02-12 16:12:45,665 INFO assembly.AggregateBy (AggregateBy.java:removeEldestEntry(333)) - total mem is 46% of max mem, to better utilize unused memory consider increasing current LRU threshold with system property "cascading.aggregateby.threshold"
2015-02-12 16:12:45,666 INFO assembly.AggregateBy (AggregateBy.java:removeEldestEntry(327)) - flushed keys num times: 7, with threshold: 2
2015-02-12 16:12:45,666 INFO assembly.AggregateBy (AggregateBy.java:removeEldestEntry(328)) - mem on flush (mb), free: 197, total: 310, max: 672
2015-02-12 16:12:45,666 INFO assembly.AggregateBy (AggregateBy.java:removeEldestEntry(333)) - total mem is 46% of max mem, to better utilize unused memory consider increasing current LRU threshold with system property "cascading.aggregateby.threshold"
2015-02-12 16:12:45,667 INFO mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) -
2015-02-12 16:12:45,667 INFO mapred.MapTask (MapTask.java:flush(1457)) - Starting flush of map output
2015-02-12 16:12:45,667 INFO mapred.MapTask (MapTask.java:flush(1475)) - Spilling map output
2015-02-12 16:12:45,667 INFO mapred.MapTask (MapTask.java:flush(1476)) - bufstart = 0; bufend = 216; bufvoid = 104857600
2015-02-12 16:12:45,667 INFO mapred.MapTask (MapTask.java:flush(1478)) - kvstart = 26214396(104857584); kvend = 26214364(104857456); length = 33/6553600
2015-02-12 16:12:45,670 INFO mapred.MapTask (MapTask.java:sortAndSpill(1660)) - Finished spill 0
2015-02-12 16:12:45,672 INFO mapred.Task (Task.java:done(1001)) - Task:attempt_local1069201491_0001_m_000001_0 is done. And is in the process of committing
2015-02-12 16:12:45,674 INFO mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) - file:/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/classes/test/data/rhs.txt:0+51
2015-02-12 16:12:45,674 INFO mapred.Task (Task.java:sendDone(1121)) - Task 'attempt_local1069201491_0001_m_000001_0' done.
2015-02-12 16:12:45,674 INFO mapred.LocalJobRunner (LocalJobRunner.java:run(249)) - Finishing task: attempt_local1069201491_0001_m_000001_0
2015-02-12 16:12:45,675 INFO mapred.LocalJobRunner (LocalJobRunner.java:runTasks(456)) - map task executor complete.
2015-02-12 16:12:45,677 INFO mapred.LocalJobRunner (LocalJobRunner.java:runTasks(448)) - Waiting for reduce tasks
2015-02-12 16:12:45,677 INFO mapred.LocalJobRunner (LocalJobRunner.java:run(302)) - Starting task: attempt_local1069201491_0001_r_000000_0
2015-02-12 16:12:45,682 INFO util.ProcfsBasedProcessTree (ProcfsBasedProcessTree.java:isAvailable(181)) - ProcfsBasedProcessTree currently is supported only on Linux.
2015-02-12 16:12:45,682 INFO mapred.Task (Task.java:initialize(587)) - Using ResourceCalculatorProcessTree : null
2015-02-12 16:12:45,684 INFO mapred.ReduceTask (ReduceTask.java:run(362)) - Using ShuffleConsumerPlugin: org.apache.hadoop.mapreduce.task.reduce.Shuffle@1cd459c4
2015-02-12 16:12:45,691 INFO reduce.MergeManagerImpl (MergeManagerImpl.java:<init>(195)) - MergerManager: memoryLimit=493250144, maxSingleShuffleLimit=123312536, mergeThreshold=325545120, ioSortFactor=10, memToMemMergeOutputsThreshold=10
2015-02-12 16:12:45,693 INFO reduce.EventFetcher (EventFetcher.java:run(61)) - attempt_local1069201491_0001_r_000000_0 Thread started: EventFetcher for fetching Map Completion Events
2015-02-12 16:12:45,709 INFO reduce.LocalFetcher (LocalFetcher.java:copyMapOutput(141)) - localfetcher#1 about to shuffle output of map attempt_local1069201491_0001_m_000001_0 decomp: 236 len: 240 to MEMORY
2015-02-12 16:12:45,711 INFO reduce.InMemoryMapOutput (InMemoryMapOutput.java:shuffle(100)) - Read 236 bytes from map-output for attempt_local1069201491_0001_m_000001_0
2015-02-12 16:12:45,730 INFO reduce.MergeManagerImpl (MergeManagerImpl.java:closeInMemoryFile(313)) - closeInMemoryFile -> map-output of size: 236, inMemoryMapOutputs.size() -> 1, commitMemory -> 0, usedMemory ->236
2015-02-12 16:12:45,732 INFO reduce.LocalFetcher (LocalFetcher.java:copyMapOutput(141)) - localfetcher#1 about to shuffle output of map attempt_local1069201491_0001_m_000000_0 decomp: 236 len: 240 to MEMORY
2015-02-12 16:12:45,732 INFO reduce.InMemoryMapOutput (InMemoryMapOutput.java:shuffle(100)) - Read 236 bytes from map-output for attempt_local1069201491_0001_m_000000_0
2015-02-12 16:12:45,733 INFO reduce.MergeManagerImpl (MergeManagerImpl.java:closeInMemoryFile(313)) - closeInMemoryFile -> map-output of size: 236, inMemoryMapOutputs.size() -> 2, commitMemory -> 236, usedMemory ->472
2015-02-12 16:12:45,733 INFO reduce.EventFetcher (EventFetcher.java:run(76)) - EventFetcher is interrupted.. Returning
2015-02-12 16:12:45,734 INFO mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) - 2 / 2 copied.
2015-02-12 16:12:45,734 INFO reduce.MergeManagerImpl (MergeManagerImpl.java:finalMerge(673)) - finalMerge called with 2 in-memory map-outputs and 0 on-disk map-outputs
2015-02-12 16:12:45,739 INFO mapred.Merger (Merger.java:merge(606)) - Merging 2 sorted segments
2015-02-12 16:12:45,739 INFO mapred.Merger (Merger.java:merge(705)) - Down to the last merge-pass, with 2 segments left of total size: 454 bytes
2015-02-12 16:12:45,741 INFO reduce.MergeManagerImpl (MergeManagerImpl.java:finalMerge(750)) - Merged 2 segments, 472 bytes to disk to satisfy reduce memory limit
2015-02-12 16:12:45,742 INFO reduce.MergeManagerImpl (MergeManagerImpl.java:finalMerge(780)) - Merging 1 files, 474 bytes from disk
2015-02-12 16:12:45,742 INFO reduce.MergeManagerImpl (MergeManagerImpl.java:finalMerge(795)) - Merging 0 segments, 0 bytes from memory into reduce
2015-02-12 16:12:45,743 INFO mapred.Merger (Merger.java:merge(606)) - Merging 1 sorted segments
2015-02-12 16:12:45,743 INFO mapred.Merger (Merger.java:merge(705)) - Down to the last merge-pass, with 1 segments left of total size: 461 bytes
2015-02-12 16:12:45,744 INFO mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) - 2 / 2 copied.
2015-02-12 16:12:45,746 INFO hadoop.FlowReducer (FlowReducer.java:configure(78)) - cascading version: 2.6.3
2015-02-12 16:12:45,746 INFO hadoop.FlowReducer (FlowReducer.java:configure(79)) - child jvm opts: -Xmx200m
2015-02-12 16:12:45,761 INFO hadoop.FlowReducer (FlowReducer.java:configure(97)) - sourcing from: GroupBy(name)[by: multi-lhs:[{1}:'char'] multi-rhs:[{1}:'char']]
2015-02-12 16:12:45,761 INFO hadoop.FlowReducer (FlowReducer.java:configure(100)) - sinking to: Hfs["TextDelimited[['char', 'sum', 'count', 'average']]"]["/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/test/output/hadoop2-mr1/assemblyhelpersplatform/multimerge+false"]
2015-02-12 16:12:45,781 INFO mapred.Task (Task.java:done(1001)) - Task:attempt_local1069201491_0001_r_000000_0 is done. And is in the process of committing
2015-02-12 16:12:45,782 INFO mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) - 2 / 2 copied.
2015-02-12 16:12:45,782 INFO mapred.Task (Task.java:commit(1162)) - Task attempt_local1069201491_0001_r_000000_0 is allowed to commit now
2015-02-12 16:12:45,784 INFO output.FileOutputCommitter (FileOutputCommitter.java:commitTask(439)) - Saved output of task 'attempt_local1069201491_0001_r_000000_0' to file:/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/test/output/hadoop2-mr1/assemblyhelpersplatform/multimerge+false/_temporary/0/task_local1069201491_0001_r_000000
2015-02-12 16:12:45,785 INFO mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) - reduce > reduce
2015-02-12 16:12:45,785 INFO mapred.Task (Task.java:sendDone(1121)) - Task 'attempt_local1069201491_0001_r_000000_0' done.
2015-02-12 16:12:45,785 INFO mapred.LocalJobRunner (LocalJobRunner.java:run(325)) - Finishing task: attempt_local1069201491_0001_r_000000_0
2015-02-12 16:12:45,785 INFO mapred.LocalJobRunner (LocalJobRunner.java:runTasks(456)) - reduce task executor complete.
2015-02-12 16:12:45,811 INFO util.Hadoop18TapUtil (Hadoop18TapUtil.java:cleanTempPath(219)) - deleting temp path /Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/test/output/hadoop2-mr1/assemblyhelpersplatform/multimerge+false/_temporary
2015-02-12 16:12:45,837 INFO mapred.FileInputFormat (FileInputFormat.java:listStatus(247)) - Total input paths to process : 1
2015-02-12 16:12:45,868 INFO mapred.FileInputFormat (FileInputFormat.java:listStatus(247)) - Total input paths to process : 1
2015-02-12 16:12:45,870 INFO cascading.PlatformTestCase (PlatformTestCase.java:tearDown(209)) - copying to local /Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/test/output//hadoop2-mr1/assemblyhelpersplatform/multimerge+false
2015-02-12 16:12:45,890 INFO util.HadoopUtil (HadoopUtil.java:findMainClass(336)) - using default application jar, may cause class not found exceptions on the cluster
2015-02-12 16:12:45,891 INFO planner.HadoopPlanner (HadoopPlanner.java:initialize(225)) - using application jar: /Users/soberoi/.gradle/caches/modules-2/files-2.1/cascading/cascading-hadoop2-mr1/2.6.3/9ae016e2b8d246b9c1de9168e715d9f1b971aa4a/cascading-hadoop2-mr1-2.6.3.jar
2015-02-12 16:12:45,899 INFO Configuration.deprecation (Configuration.java:warnOnceIfDeprecated(1022)) - mapred.used.genericoptionsparser is deprecated. Instead, use mapreduce.client.genericoptionsparser.used
2015-02-12 16:12:45,902 INFO Configuration.deprecation (Configuration.java:warnOnceIfDeprecated(1022)) - mapred.job.tracker is deprecated. Instead, use mapreduce.jobtracker.address
2015-02-12 16:12:45,903 INFO Configuration.deprecation (Configuration.java:warnOnceIfDeprecated(1022)) - mapred.output.compress is deprecated. Instead, use mapreduce.output.fileoutputformat.compress
2015-02-12 16:12:45,911 INFO flow.Flow (BaseFlow.java:logInfo(1378)) - [max] starting
2015-02-12 16:12:45,911 INFO flow.Flow (BaseFlow.java:logInfo(1378)) - [max] source: Hfs["TextDelimited[['num', 'char']]"]["/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/classes/test/data/lhs.txt"]
2015-02-12 16:12:45,912 INFO flow.Flow (BaseFlow.java:logInfo(1378)) - [max] sink: Hfs["TextDelimited[['num', 'min']]"]["/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/test/output/hadoop2-mr1/assemblyhelpersplatform/minbystring"]
2015-02-12 16:12:45,912 INFO flow.Flow (BaseFlow.java:logInfo(1378)) - [max] parallel execution is enabled: false
2015-02-12 16:12:45,912 INFO flow.Flow (BaseFlow.java:logInfo(1378)) - [max] starting jobs: 1
2015-02-12 16:12:45,912 INFO flow.Flow (BaseFlow.java:logInfo(1378)) - [max] allocating threads: 1
2015-02-12 16:12:45,912 INFO flow.FlowStep (BaseFlowStep.java:logInfo(834)) - [max] starting step: (1/1) ...lpersplatform/minbystring
2015-02-12 16:12:45,913 INFO jvm.JvmMetrics (JvmMetrics.java:init(71)) - Cannot initialize JVM Metrics with processName=JobTracker, sessionId= - already initialized
2015-02-12 16:12:45,915 INFO jvm.JvmMetrics (JvmMetrics.java:init(71)) - Cannot initialize JVM Metrics with processName=JobTracker, sessionId= - already initialized
2015-02-12 16:12:45,937 INFO mapred.FileInputFormat (FileInputFormat.java:listStatus(247)) - Total input paths to process : 1
2015-02-12 16:12:45,956 INFO mapreduce.JobSubmitter (JobSubmitter.java:submitJobInternal(492)) - number of splits:1
2015-02-12 16:12:45,976 INFO mapreduce.JobSubmitter (JobSubmitter.java:printTokens(575)) - Submitting tokens for job: job_local911131477_0002
2015-02-12 16:12:45,988 WARN conf.Configuration (Configuration.java:loadProperty(2510)) - file:/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/tmp/cascading/staging/soberoi911131477/.staging/job_local911131477_0002/job.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.retry.interval; Ignoring.
2015-02-12 16:12:45,991 WARN conf.Configuration (Configuration.java:loadProperty(2510)) - file:/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/tmp/cascading/staging/soberoi911131477/.staging/job_local911131477_0002/job.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.attempts; Ignoring.
2015-02-12 16:12:46,039 WARN conf.Configuration (Configuration.java:loadProperty(2510)) - file:/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/test/tmp/mapred/local/localRunner/soberoi/job_local911131477_0002/job_local911131477_0002.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.retry.interval; Ignoring.
2015-02-12 16:12:46,041 WARN conf.Configuration (Configuration.java:loadProperty(2510)) - file:/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/test/tmp/mapred/local/localRunner/soberoi/job_local911131477_0002/job_local911131477_0002.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.attempts; Ignoring.
2015-02-12 16:12:46,042 INFO mapreduce.Job (Job.java:submit(1299)) - The url to track the job: http://localhost:8080/
2015-02-12 16:12:46,042 INFO mapred.LocalJobRunner (LocalJobRunner.java:createOutputCommitter(471)) - OutputCommitter set in config null
2015-02-12 16:12:46,042 INFO flow.FlowStep (BaseFlowStep.java:logInfo(834)) - [max] submitted hadoop job: job_local911131477_0002
2015-02-12 16:12:46,043 INFO flow.FlowStep (BaseFlowStep.java:logInfo(834)) - [max] tracking url: http://localhost:8080/
2015-02-12 16:12:46,042 INFO mapred.LocalJobRunner (LocalJobRunner.java:createOutputCommitter(489)) - OutputCommitter is org.apache.hadoop.mapred.FileOutputCommitter
2015-02-12 16:12:46,045 INFO mapred.LocalJobRunner (LocalJobRunner.java:runTasks(448)) - Waiting for map tasks
2015-02-12 16:12:46,046 INFO mapred.LocalJobRunner (LocalJobRunner.java:run(224)) - Starting task: attempt_local911131477_0002_m_000000_0
2015-02-12 16:12:46,047 INFO util.ProcfsBasedProcessTree (ProcfsBasedProcessTree.java:isAvailable(181)) - ProcfsBasedProcessTree currently is supported only on Linux.
2015-02-12 16:12:46,047 INFO mapred.Task (Task.java:initialize(587)) - Using ResourceCalculatorProcessTree : null
2015-02-12 16:12:46,049 INFO io.MultiInputSplit (MultiInputSplit.java:readFields(161)) - current split input path: file:/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/classes/test/data/lhs.txt
2015-02-12 16:12:46,049 INFO mapred.MapTask (MapTask.java:updateJobWithSplit(482)) - Processing split: cascading.tap.hadoop.io.MultiInputSplit@3bf9fe08
2015-02-12 16:12:46,051 INFO mapred.MapTask (MapTask.java:runOldMapper(436)) - numReduceTasks: 1
2015-02-12 16:12:46,104 INFO mapred.MapTask (MapTask.java:setEquator(1202)) - (EQUATOR) 0 kvi 26214396(104857584)
2015-02-12 16:12:46,104 INFO mapred.MapTask (MapTask.java:init(995)) - mapreduce.task.io.sort.mb: 100
2015-02-12 16:12:46,104 INFO mapred.MapTask (MapTask.java:init(996)) - soft limit at 83886080
2015-02-12 16:12:46,104 INFO mapred.MapTask (MapTask.java:init(997)) - bufstart = 0; bufvoid = 104857600
2015-02-12 16:12:46,104 INFO mapred.MapTask (MapTask.java:init(998)) - kvstart = 26214396; length = 6553600
2015-02-12 16:12:46,106 INFO mapred.MapTask (MapTask.java:createSortingCollector(402)) - Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer
2015-02-12 16:12:46,106 INFO hadoop.FlowMapper (FlowMapper.java:configure(71)) - cascading version: 2.6.3
2015-02-12 16:12:46,106 INFO hadoop.FlowMapper (FlowMapper.java:configure(72)) - child jvm opts: -Xmx200m
2015-02-12 16:12:46,113 INFO hadoop.FlowMapper (FlowMapper.java:configure(87)) - sourcing from: Hfs["TextDelimited[['num', 'char']]"]["/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/classes/test/data/lhs.txt"]
2015-02-12 16:12:46,113 INFO hadoop.FlowMapper (FlowMapper.java:configure(90)) - sinking to: GroupBy(max)[by:[{1}:'num']]
2015-02-12 16:12:46,113 INFO assembly.AggregateBy (AggregateBy.java:prepare(267)) - using threshold value: 2
2015-02-12 16:12:46,115 INFO assembly.AggregateBy (AggregateBy.java:removeEldestEntry(327)) - flushed keys num times: 1, with threshold: 2
2015-02-12 16:12:46,115 INFO assembly.AggregateBy (AggregateBy.java:removeEldestEntry(328)) - mem on flush (mb), free: 129, total: 410, max: 672
2015-02-12 16:12:46,115 INFO assembly.AggregateBy (AggregateBy.java:removeEldestEntry(333)) - total mem is 61% of max mem, to better utilize unused memory consider increasing current LRU threshold with system property "cascading.aggregateby.threshold"
2015-02-12 16:12:46,115 INFO assembly.AggregateBy (AggregateBy.java:removeEldestEntry(327)) - flushed keys num times: 3, with threshold: 2
2015-02-12 16:12:46,115 INFO assembly.AggregateBy (AggregateBy.java:removeEldestEntry(328)) - mem on flush (mb), free: 129, total: 410, max: 672
2015-02-12 16:12:46,115 INFO assembly.AggregateBy (AggregateBy.java:removeEldestEntry(333)) - total mem is 61% of max mem, to better utilize unused memory consider increasing current LRU threshold with system property "cascading.aggregateby.threshold"
2015-02-12 16:12:46,116 INFO mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) -
2015-02-12 16:12:46,117 INFO mapred.MapTask (MapTask.java:flush(1457)) - Starting flush of map output
2015-02-12 16:12:46,117 INFO mapred.MapTask (MapTask.java:flush(1475)) - Spilling map output
2015-02-12 16:12:46,117 INFO mapred.MapTask (MapTask.java:flush(1476)) - bufstart = 0; bufend = 75; bufvoid = 104857600
2015-02-12 16:12:46,117 INFO mapred.MapTask (MapTask.java:flush(1478)) - kvstart = 26214396(104857584); kvend = 26214380(104857520); length = 17/6553600
2015-02-12 16:12:46,119 INFO mapred.MapTask (MapTask.java:sortAndSpill(1660)) - Finished spill 0
2015-02-12 16:12:46,120 INFO mapred.Task (Task.java:done(1001)) - Task:attempt_local911131477_0002_m_000000_0 is done. And is in the process of committing
2015-02-12 16:12:46,122 INFO mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) - file:/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/classes/test/data/lhs.txt:0+52
2015-02-12 16:12:46,122 INFO mapred.Task (Task.java:sendDone(1121)) - Task 'attempt_local911131477_0002_m_000000_0' done.
2015-02-12 16:12:46,122 INFO mapred.LocalJobRunner (LocalJobRunner.java:run(249)) - Finishing task: attempt_local911131477_0002_m_000000_0
2015-02-12 16:12:46,122 INFO mapred.LocalJobRunner (LocalJobRunner.java:runTasks(456)) - map task executor complete.
2015-02-12 16:12:46,123 INFO mapred.LocalJobRunner (LocalJobRunner.java:runTasks(448)) - Waiting for reduce tasks
2015-02-12 16:12:46,123 INFO mapred.LocalJobRunner (LocalJobRunner.java:run(302)) - Starting task: attempt_local911131477_0002_r_000000_0
2015-02-12 16:12:46,124 INFO util.ProcfsBasedProcessTree (ProcfsBasedProcessTree.java:isAvailable(181)) - ProcfsBasedProcessTree currently is supported only on Linux.
2015-02-12 16:12:46,124 INFO mapred.Task (Task.java:initialize(587)) - Using ResourceCalculatorProcessTree : null
2015-02-12 16:12:46,124 INFO mapred.ReduceTask (ReduceTask.java:run(362)) - Using ShuffleConsumerPlugin: org.apache.hadoop.mapreduce.task.reduce.Shuffle@3a2d2d6c
2015-02-12 16:12:46,125 INFO reduce.MergeManagerImpl (MergeManagerImpl.java:<init>(195)) - MergerManager: memoryLimit=493250144, maxSingleShuffleLimit=123312536, mergeThreshold=325545120, ioSortFactor=10, memToMemMergeOutputsThreshold=10
2015-02-12 16:12:46,126 INFO reduce.EventFetcher (EventFetcher.java:run(61)) - attempt_local911131477_0002_r_000000_0 Thread started: EventFetcher for fetching Map Completion Events
2015-02-12 16:12:46,126 INFO reduce.LocalFetcher (LocalFetcher.java:copyMapOutput(141)) - localfetcher#2 about to shuffle output of map attempt_local911131477_0002_m_000000_0 decomp: 87 len: 91 to MEMORY
2015-02-12 16:12:46,127 INFO reduce.InMemoryMapOutput (InMemoryMapOutput.java:shuffle(100)) - Read 87 bytes from map-output for attempt_local911131477_0002_m_000000_0
2015-02-12 16:12:46,127 INFO reduce.MergeManagerImpl (MergeManagerImpl.java:closeInMemoryFile(313)) - closeInMemoryFile -> map-output of size: 87, inMemoryMapOutputs.size() -> 1, commitMemory -> 0, usedMemory ->87
2015-02-12 16:12:46,127 INFO reduce.EventFetcher (EventFetcher.java:run(76)) - EventFetcher is interrupted.. Returning
2015-02-12 16:12:46,128 INFO mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) - 1 / 1 copied.
2015-02-12 16:12:46,128 INFO reduce.MergeManagerImpl (MergeManagerImpl.java:finalMerge(673)) - finalMerge called with 1 in-memory map-outputs and 0 on-disk map-outputs
2015-02-12 16:12:46,130 INFO mapred.Merger (Merger.java:merge(606)) - Merging 1 sorted segments
2015-02-12 16:12:46,130 INFO mapred.Merger (Merger.java:merge(705)) - Down to the last merge-pass, with 1 segments left of total size: 78 bytes
2015-02-12 16:12:46,132 INFO reduce.MergeManagerImpl (MergeManagerImpl.java:finalMerge(750)) - Merged 1 segments, 87 bytes to disk to satisfy reduce memory limit
2015-02-12 16:12:46,132 INFO reduce.MergeManagerImpl (MergeManagerImpl.java:finalMerge(780)) - Merging 1 files, 91 bytes from disk
2015-02-12 16:12:46,132 INFO reduce.MergeManagerImpl (MergeManagerImpl.java:finalMerge(795)) - Merging 0 segments, 0 bytes from memory into reduce
2015-02-12 16:12:46,132 INFO mapred.Merger (Merger.java:merge(606)) - Merging 1 sorted segments
2015-02-12 16:12:46,132 INFO mapred.Merger (Merger.java:merge(705)) - Down to the last merge-pass, with 1 segments left of total size: 78 bytes
2015-02-12 16:12:46,133 INFO mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) - 1 / 1 copied.
2015-02-12 16:12:46,133 INFO hadoop.FlowReducer (FlowReducer.java:configure(78)) - cascading version: 2.6.3
2015-02-12 16:12:46,134 INFO hadoop.FlowReducer (FlowReducer.java:configure(79)) - child jvm opts: -Xmx200m
2015-02-12 16:12:46,139 INFO hadoop.FlowReducer (FlowReducer.java:configure(97)) - sourcing from: GroupBy(max)[by:[{1}:'num']]
2015-02-12 16:12:46,140 INFO hadoop.FlowReducer (FlowReducer.java:configure(100)) - sinking to: Hfs["TextDelimited[['num', 'min']]"]["/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/test/output/hadoop2-mr1/assemblyhelpersplatform/minbystring"]
2015-02-12 16:12:46,149 INFO mapred.Task (Task.java:done(1001)) - Task:attempt_local911131477_0002_r_000000_0 is done. And is in the process of committing
2015-02-12 16:12:46,150 INFO mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) - 1 / 1 copied.
2015-02-12 16:12:46,150 INFO mapred.Task (Task.java:commit(1162)) - Task attempt_local911131477_0002_r_000000_0 is allowed to commit now
2015-02-12 16:12:46,151 INFO output.FileOutputCommitter (FileOutputCommitter.java:commitTask(439)) - Saved output of task 'attempt_local911131477_0002_r_000000_0' to file:/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/test/output/hadoop2-mr1/assemblyhelpersplatform/minbystring/_temporary/0/task_local911131477_0002_r_000000
2015-02-12 16:12:46,152 INFO mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) - reduce > reduce
2015-02-12 16:12:46,152 INFO mapred.Task (Task.java:sendDone(1121)) - Task 'attempt_local911131477_0002_r_000000_0' done.
2015-02-12 16:12:46,152 INFO mapred.LocalJobRunner (LocalJobRunner.java:run(325)) - Finishing task: attempt_local911131477_0002_r_000000_0
2015-02-12 16:12:46,152 INFO mapred.LocalJobRunner (LocalJobRunner.java:runTasks(456)) - reduce task executor complete.
2015-02-12 16:12:46,170 INFO util.Hadoop18TapUtil (Hadoop18TapUtil.java:cleanTempPath(219)) - deleting temp path /Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/test/output/hadoop2-mr1/assemblyhelpersplatform/minbystring/_temporary
2015-02-12 16:12:46,203 INFO mapred.FileInputFormat (FileInputFormat.java:listStatus(247)) - Total input paths to process : 1
2015-02-12 16:12:46,230 INFO mapred.FileInputFormat (FileInputFormat.java:listStatus(247)) - Total input paths to process : 1
2015-02-12 16:12:46,231 INFO cascading.PlatformTestCase (PlatformTestCase.java:tearDown(209)) - copying to local /Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/test/output//hadoop2-mr1/assemblyhelpersplatform/minbystring
2015-02-12 16:12:46,259 INFO util.HadoopUtil (HadoopUtil.java:findMainClass(336)) - using default application jar, may cause class not found exceptions on the cluster
2015-02-12 16:12:46,260 INFO planner.HadoopPlanner (HadoopPlanner.java:initialize(225)) - using application jar: /Users/soberoi/.gradle/caches/modules-2/files-2.1/cascading/cascading-hadoop2-mr1/2.6.3/9ae016e2b8d246b9c1de9168e715d9f1b971aa4a/cascading-hadoop2-mr1-2.6.3.jar
2015-02-12 16:12:46,275 INFO Configuration.deprecation (Configuration.java:warnOnceIfDeprecated(1022)) - mapred.used.genericoptionsparser is deprecated. Instead, use mapreduce.client.genericoptionsparser.used
2015-02-12 16:12:46,282 INFO Configuration.deprecation (Configuration.java:warnOnceIfDeprecated(1022)) - mapred.job.tracker is deprecated. Instead, use mapreduce.jobtracker.address
2015-02-12 16:12:46,283 INFO Configuration.deprecation (Configuration.java:warnOnceIfDeprecated(1022)) - mapred.output.compress is deprecated. Instead, use mapreduce.output.fileoutputformat.compress
2015-02-12 16:12:46,294 INFO flow.Flow (BaseFlow.java:logInfo(1378)) - [average-lhs+average-rhs] starting
2015-02-12 16:12:46,295 INFO flow.Flow (BaseFlow.java:logInfo(1378)) - [average-lhs+average-rhs] source: Hfs["TextDelimited[['num', 'char']]"]["/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/classes/test/data/rhs.txt"]
2015-02-12 16:12:46,295 INFO flow.Flow (BaseFlow.java:logInfo(1378)) - [average-lhs+average-rhs] source: Hfs["TextDelimited[['num', 'char']]"]["/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/classes/test/data/lhs.txt"]
2015-02-12 16:12:46,295 INFO flow.Flow (BaseFlow.java:logInfo(1378)) - [average-lhs+average-rhs] sink: Hfs["TextDelimited[['char', 'average']]"]["/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/test/output/hadoop2-mr1/assemblyhelpersplatform/mergeaverage"]
2015-02-12 16:12:46,295 INFO flow.Flow (BaseFlow.java:logInfo(1378)) - [average-lhs+average-rhs] parallel execution is enabled: false
2015-02-12 16:12:46,295 INFO flow.Flow (BaseFlow.java:logInfo(1378)) - [average-lhs+average-rhs] starting jobs: 1
2015-02-12 16:12:46,295 INFO flow.Flow (BaseFlow.java:logInfo(1378)) - [average-lhs+average-rhs] allocating threads: 1
2015-02-12 16:12:46,296 INFO flow.FlowStep (BaseFlowStep.java:logInfo(834)) - [average-lhs+average-rhs] starting step: (1/1) ...persplatform/mergeaverage
2015-02-12 16:12:46,297 INFO jvm.JvmMetrics (JvmMetrics.java:init(71)) - Cannot initialize JVM Metrics with processName=JobTracker, sessionId= - already initialized
2015-02-12 16:12:46,298 INFO jvm.JvmMetrics (JvmMetrics.java:init(71)) - Cannot initialize JVM Metrics with processName=JobTracker, sessionId= - already initialized
2015-02-12 16:12:46,328 INFO mapred.FileInputFormat (FileInputFormat.java:listStatus(247)) - Total input paths to process : 1
2015-02-12 16:12:46,329 INFO mapred.FileInputFormat (FileInputFormat.java:listStatus(247)) - Total input paths to process : 1
2015-02-12 16:12:46,352 INFO mapreduce.JobSubmitter (JobSubmitter.java:submitJobInternal(492)) - number of splits:2
2015-02-12 16:12:46,417 INFO mapreduce.JobSubmitter (JobSubmitter.java:printTokens(575)) - Submitting tokens for job: job_local1056486563_0003
2015-02-12 16:12:46,436 WARN conf.Configuration (Configuration.java:loadProperty(2510)) - file:/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/tmp/cascading/staging/soberoi1056486563/.staging/job_local1056486563_0003/job.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.retry.interval; Ignoring.
2015-02-12 16:12:46,438 WARN conf.Configuration (Configuration.java:loadProperty(2510)) - file:/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/tmp/cascading/staging/soberoi1056486563/.staging/job_local1056486563_0003/job.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.attempts; Ignoring.
2015-02-12 16:12:46,535 WARN conf.Configuration (Configuration.java:loadProperty(2510)) - file:/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/test/tmp/mapred/local/localRunner/soberoi/job_local1056486563_0003/job_local1056486563_0003.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.retry.interval; Ignoring.
2015-02-12 16:12:46,536 WARN conf.Configuration (Configuration.java:loadProperty(2510)) - file:/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/test/tmp/mapred/local/localRunner/soberoi/job_local1056486563_0003/job_local1056486563_0003.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.attempts; Ignoring.
2015-02-12 16:12:46,537 INFO mapreduce.Job (Job.java:submit(1299)) - The url to track the job: http://localhost:8080/
2015-02-12 16:12:46,537 INFO mapred.LocalJobRunner (LocalJobRunner.java:createOutputCommitter(471)) - OutputCommitter set in config null
2015-02-12 16:12:46,537 INFO flow.FlowStep (BaseFlowStep.java:logInfo(834)) - [average-lhs+average-rhs] submitted hadoop job: job_local1056486563_0003
2015-02-12 16:12:46,537 INFO mapred.LocalJobRunner (LocalJobRunner.java:createOutputCommitter(489)) - OutputCommitter is org.apache.hadoop.mapred.FileOutputCommitter
2015-02-12 16:12:46,537 INFO flow.FlowStep (BaseFlowStep.java:logInfo(834)) - [average-lhs+average-rhs] tracking url: http://localhost:8080/
2015-02-12 16:12:46,539 INFO mapred.LocalJobRunner (LocalJobRunner.java:runTasks(448)) - Waiting for map tasks
2015-02-12 16:12:46,539 INFO mapred.LocalJobRunner (LocalJobRunner.java:run(224)) - Starting task: attempt_local1056486563_0003_m_000000_0
2015-02-12 16:12:46,541 INFO util.ProcfsBasedProcessTree (ProcfsBasedProcessTree.java:isAvailable(181)) - ProcfsBasedProcessTree currently is supported only on Linux.
2015-02-12 16:12:46,541 INFO mapred.Task (Task.java:initialize(587)) - Using ResourceCalculatorProcessTree : null
2015-02-12 16:12:46,542 INFO io.MultiInputSplit (MultiInputSplit.java:readFields(161)) - current split input path: file:/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/classes/test/data/lhs.txt
2015-02-12 16:12:46,542 INFO mapred.MapTask (MapTask.java:updateJobWithSplit(482)) - Processing split: cascading.tap.hadoop.io.MultiInputSplit@295d5fc0
2015-02-12 16:12:46,543 INFO mapred.MapTask (MapTask.java:runOldMapper(436)) - numReduceTasks: 1
2015-02-12 16:12:46,555 INFO mapred.MapTask (MapTask.java:setEquator(1202)) - (EQUATOR) 0 kvi 26214396(104857584)
2015-02-12 16:12:46,555 INFO mapred.MapTask (MapTask.java:init(995)) - mapreduce.task.io.sort.mb: 100
2015-02-12 16:12:46,555 INFO mapred.MapTask (MapTask.java:init(996)) - soft limit at 83886080
2015-02-12 16:12:46,555 INFO mapred.MapTask (MapTask.java:init(997)) - bufstart = 0; bufvoid = 104857600
2015-02-12 16:12:46,555 INFO mapred.MapTask (MapTask.java:init(998)) - kvstart = 26214396; length = 6553600
2015-02-12 16:12:46,557 INFO mapred.MapTask (MapTask.java:createSortingCollector(402)) - Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer
2015-02-12 16:12:46,557 INFO hadoop.FlowMapper (FlowMapper.java:configure(71)) - cascading version: 2.6.3
2015-02-12 16:12:46,557 INFO hadoop.FlowMapper (FlowMapper.java:configure(72)) - child jvm opts: -Xmx200m
2015-02-12 16:12:46,567 INFO hadoop.FlowMapper (FlowMapper.java:configure(87)) - sourcing from: Hfs["TextDelimited[['num', 'char']]"]["/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/classes/test/data/lhs.txt"]
2015-02-12 16:12:46,567 INFO hadoop.FlowMapper (FlowMapper.java:configure(90)) - sinking to: GroupBy(average-lhs+average-rhs)[by: average-lhs:[{1}:'char'] average-rhs:[{1}:'char']]
2015-02-12 16:12:46,567 INFO assembly.AggregateBy (AggregateBy.java:prepare(267)) - using threshold value: 2
2015-02-12 16:12:46,569 INFO assembly.AggregateBy (AggregateBy.java:removeEldestEntry(327)) - flushed keys num times: 1, with threshold: 2
2015-02-12 16:12:46,569 INFO assembly.AggregateBy (AggregateBy.java:removeEldestEntry(328)) - mem on flush (mb), free: 193, total: 512, max: 672
2015-02-12 16:12:46,569 INFO assembly.AggregateBy (AggregateBy.java:removeEldestEntry(333)) - total mem is 76% of max mem, to better utilize unused memory consider increasing current LRU threshold with system property "cascading.aggregateby.threshold"
2015-02-12 16:12:46,569 INFO assembly.AggregateBy (AggregateBy.java:removeEldestEntry(327)) - flushed keys num times: 3, with threshold: 2
2015-02-12 16:12:46,570 INFO assembly.AggregateBy (AggregateBy.java:removeEldestEntry(328)) - mem on flush (mb), free: 193, total: 512, max: 672
2015-02-12 16:12:46,570 INFO assembly.AggregateBy (AggregateBy.java:removeEldestEntry(333)) - total mem is 76% of max mem, to better utilize unused memory consider increasing current LRU threshold with system property "cascading.aggregateby.threshold"
2015-02-12 16:12:46,570 INFO assembly.AggregateBy (AggregateBy.java:removeEldestEntry(327)) - flushed keys num times: 5, with threshold: 2
2015-02-12 16:12:46,570 INFO assembly.AggregateBy (AggregateBy.java:removeEldestEntry(328)) - mem on flush (mb), free: 193, total: 512, max: 672
2015-02-12 16:12:46,570 INFO assembly.AggregateBy (AggregateBy.java:removeEldestEntry(333)) - total mem is 76% of max mem, to better utilize unused memory consider increasing current LRU threshold with system property "cascading.aggregateby.threshold"
2015-02-12 16:12:46,571 INFO assembly.AggregateBy (AggregateBy.java:removeEldestEntry(327)) - flushed keys num times: 7, with threshold: 2
2015-02-12 16:12:46,571 INFO assembly.AggregateBy (AggregateBy.java:removeEldestEntry(328)) - mem on flush (mb), free: 193, total: 512, max: 672
2015-02-12 16:12:46,571 INFO assembly.AggregateBy (AggregateBy.java:removeEldestEntry(333)) - total mem is 76% of max mem, to better utilize unused memory consider increasing current LRU threshold with system property "cascading.aggregateby.threshold"
2015-02-12 16:12:46,572 INFO mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) -
2015-02-12 16:12:46,572 INFO mapred.MapTask (MapTask.java:flush(1457)) - Starting flush of map output
2015-02-12 16:12:46,573 INFO mapred.MapTask (MapTask.java:flush(1475)) - Spilling map output
2015-02-12 16:12:46,573 INFO mapred.MapTask (MapTask.java:flush(1476)) - bufstart = 0; bufend = 180; bufvoid = 104857600
2015-02-12 16:12:46,573 INFO mapred.MapTask (MapTask.java:flush(1478)) - kvstart = 26214396(104857584); kvend = 26214364(104857456); length = 33/6553600
2015-02-12 16:12:46,576 INFO mapred.MapTask (MapTask.java:sortAndSpill(1660)) - Finished spill 0
2015-02-12 16:12:46,578 INFO mapred.Task (Task.java:done(1001)) - Task:attempt_local1056486563_0003_m_000000_0 is done. And is in the process of committing
2015-02-12 16:12:46,580 INFO mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) - file:/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/classes/test/data/lhs.txt:0+52
2015-02-12 16:12:46,580 INFO mapred.Task (Task.java:sendDone(1121)) - Task 'attempt_local1056486563_0003_m_000000_0' done.
2015-02-12 16:12:46,580 INFO mapred.LocalJobRunner (LocalJobRunner.java:run(249)) - Finishing task: attempt_local1056486563_0003_m_000000_0
2015-02-12 16:12:46,580 INFO mapred.LocalJobRunner (LocalJobRunner.java:run(224)) - Starting task: attempt_local1056486563_0003_m_000001_0
2015-02-12 16:12:46,582 INFO util.ProcfsBasedProcessTree (ProcfsBasedProcessTree.java:isAvailable(181)) - ProcfsBasedProcessTree currently is supported only on Linux.
2015-02-12 16:12:46,582 INFO mapred.Task (Task.java:initialize(587)) - Using ResourceCalculatorProcessTree : null
2015-02-12 16:12:46,583 INFO io.MultiInputSplit (MultiInputSplit.java:readFields(161)) - current split input path: file:/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/classes/test/data/rhs.txt
2015-02-12 16:12:46,584 INFO mapred.MapTask (MapTask.java:updateJobWithSplit(482)) - Processing split: cascading.tap.hadoop.io.MultiInputSplit@43cefb4d
2015-02-12 16:12:46,585 INFO mapred.MapTask (MapTask.java:runOldMapper(436)) - numReduceTasks: 1
2015-02-12 16:12:46,623 INFO mapred.MapTask (MapTask.java:setEquator(1202)) - (EQUATOR) 0 kvi 26214396(104857584)
2015-02-12 16:12:46,623 INFO mapred.MapTask (MapTask.java:init(995)) - mapreduce.task.io.sort.mb: 100
2015-02-12 16:12:46,624 INFO mapred.MapTask (MapTask.java:init(996)) - soft limit at 83886080
2015-02-12 16:12:46,624 INFO mapred.MapTask (MapTask.java:init(997)) - bufstart = 0; bufvoid = 104857600
2015-02-12 16:12:46,624 INFO mapred.MapTask (MapTask.java:init(998)) - kvstart = 26214396; length = 6553600
2015-02-12 16:12:46,625 INFO mapred.MapTask (MapTask.java:createSortingCollector(402)) - Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer
2015-02-12 16:12:46,625 INFO hadoop.FlowMapper (FlowMapper.java:configure(71)) - cascading version: 2.6.3
2015-02-12 16:12:46,625 INFO hadoop.FlowMapper (FlowMapper.java:configure(72)) - child jvm opts: -Xmx200m
2015-02-12 16:12:46,635 INFO hadoop.FlowMapper (FlowMapper.java:configure(87)) - sourcing from: Hfs["TextDelimited[['num', 'char']]"]["/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/classes/test/data/rhs.txt"]
2015-02-12 16:12:46,635 INFO hadoop.FlowMapper (FlowMapper.java:configure(90)) - sinking to: GroupBy(average-lhs+average-rhs)[by: average-lhs:[{1}:'char'] average-rhs:[{1}:'char']]
2015-02-12 16:12:46,635 INFO assembly.AggregateBy (AggregateBy.java:prepare(267)) - using threshold value: 2
2015-02-12 16:12:46,638 INFO assembly.AggregateBy (AggregateBy.java:removeEldestEntry(327)) - flushed keys num times: 1, with threshold: 2
2015-02-12 16:12:46,638 INFO assembly.AggregateBy (AggregateBy.java:removeEldestEntry(328)) - mem on flush (mb), free: 91, total: 512, max: 672
2015-02-12 16:12:46,638 INFO assembly.AggregateBy (AggregateBy.java:removeEldestEntry(333)) - total mem is 76% of max mem, to better utilize unused memory consider increasing current LRU threshold with system property "cascading.aggregateby.threshold"
2015-02-12 16:12:46,639 INFO assembly.AggregateBy (AggregateBy.java:removeEldestEntry(327)) - flushed keys num times: 3, with threshold: 2
2015-02-12 16:12:46,639 INFO assembly.AggregateBy (AggregateBy.java:removeEldestEntry(328)) - mem on flush (mb), free: 91, total: 512, max: 672
2015-02-12 16:12:46,639 INFO assembly.AggregateBy (AggregateBy.java:removeEldestEntry(333)) - total mem is 76% of max mem, to better utilize unused memory consider increasing current LRU threshold with system property "cascading.aggregateby.threshold"
2015-02-12 16:12:46,640 INFO assembly.AggregateBy (AggregateBy.java:removeEldestEntry(327)) - flushed keys num times: 5, with threshold: 2
2015-02-12 16:12:46,640 INFO assembly.AggregateBy (AggregateBy.java:removeEldestEntry(328)) - mem on flush (mb), free: 91, total: 512, max: 672
2015-02-12 16:12:46,640 INFO assembly.AggregateBy (AggregateBy.java:removeEldestEntry(333)) - total mem is 76% of max mem, to better utilize unused memory consider increasing current LRU threshold with system property "cascading.aggregateby.threshold"
2015-02-12 16:12:46,641 INFO assembly.AggregateBy (AggregateBy.java:removeEldestEntry(327)) - flushed keys num times: 7, with threshold: 2
2015-02-12 16:12:46,641 INFO assembly.AggregateBy (AggregateBy.java:removeEldestEntry(328)) - mem on flush (mb), free: 91, total: 512, max: 672
2015-02-12 16:12:46,641 INFO assembly.AggregateBy (AggregateBy.java:removeEldestEntry(333)) - total mem is 76% of max mem, to better utilize unused memory consider increasing current LRU threshold with system property "cascading.aggregateby.threshold"
2015-02-12 16:12:46,642 INFO mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) -
2015-02-12 16:12:46,642 INFO mapred.MapTask (MapTask.java:flush(1457)) - Starting flush of map output
2015-02-12 16:12:46,642 INFO mapred.MapTask (MapTask.java:flush(1475)) - Spilling map output
2015-02-12 16:12:46,642 INFO mapred.MapTask (MapTask.java:flush(1476)) - bufstart = 0; bufend = 180; bufvoid = 104857600
2015-02-12 16:12:46,642 INFO mapred.MapTask (MapTask.java:flush(1478)) - kvstart = 26214396(104857584); kvend = 26214364(104857456); length = 33/6553600
2015-02-12 16:12:46,646 INFO mapred.MapTask (MapTask.java:sortAndSpill(1660)) - Finished spill 0
2015-02-12 16:12:46,648 INFO mapred.Task (Task.java:done(1001)) - Task:attempt_local1056486563_0003_m_000001_0 is done. And is in the process of committing
2015-02-12 16:12:46,650 INFO mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) - file:/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/classes/test/data/rhs.txt:0+51
2015-02-12 16:12:46,650 INFO mapred.Task (Task.java:sendDone(1121)) - Task 'attempt_local1056486563_0003_m_000001_0' done.
2015-02-12 16:12:46,650 INFO mapred.LocalJobRunner (LocalJobRunner.java:run(249)) - Finishing task: attempt_local1056486563_0003_m_000001_0
2015-02-12 16:12:46,650 INFO mapred.LocalJobRunner (LocalJobRunner.java:runTasks(456)) - map task executor complete.
2015-02-12 16:12:46,650 INFO mapred.LocalJobRunner (LocalJobRunner.java:runTasks(448)) - Waiting for reduce tasks
2015-02-12 16:12:46,651 INFO mapred.LocalJobRunner (LocalJobRunner.java:run(302)) - Starting task: attempt_local1056486563_0003_r_000000_0
2015-02-12 16:12:46,652 INFO util.ProcfsBasedProcessTree (ProcfsBasedProcessTree.java:isAvailable(181)) - ProcfsBasedProcessTree currently is supported only on Linux.
2015-02-12 16:12:46,652 INFO mapred.Task (Task.java:initialize(587)) - Using ResourceCalculatorProcessTree : null
2015-02-12 16:12:46,652 INFO mapred.ReduceTask (ReduceTask.java:run(362)) - Using ShuffleConsumerPlugin: org.apache.hadoop.mapreduce.task.reduce.Shuffle@5c4e4be3
2015-02-12 16:12:46,653 INFO reduce.MergeManagerImpl (MergeManagerImpl.java:<init>(195)) - MergerManager: memoryLimit=493250144, maxSingleShuffleLimit=123312536, mergeThreshold=325545120, ioSortFactor=10, memToMemMergeOutputsThreshold=10
2015-02-12 16:12:46,653 INFO reduce.EventFetcher (EventFetcher.java:run(61)) - attempt_local1056486563_0003_r_000000_0 Thread started: EventFetcher for fetching Map Completion Events
2015-02-12 16:12:46,655 INFO reduce.LocalFetcher (LocalFetcher.java:copyMapOutput(141)) - localfetcher#3 about to shuffle output of map attempt_local1056486563_0003_m_000001_0 decomp: 200 len: 204 to MEMORY
2015-02-12 16:12:46,655 INFO reduce.InMemoryMapOutput (InMemoryMapOutput.java:shuffle(100)) - Read 200 bytes from map-output for attempt_local1056486563_0003_m_000001_0
2015-02-12 16:12:46,655 INFO reduce.MergeManagerImpl (MergeManagerImpl.java:closeInMemoryFile(313)) - closeInMemoryFile -> map-output of size: 200, inMemoryMapOutputs.size() -> 1, commitMemory -> 0, usedMemory ->200
2015-02-12 16:12:46,657 INFO reduce.LocalFetcher (LocalFetcher.java:copyMapOutput(141)) - localfetcher#3 about to shuffle output of map attempt_local1056486563_0003_m_000000_0 decomp: 200 len: 204 to MEMORY
2015-02-12 16:12:46,658 INFO reduce.InMemoryMapOutput (InMemoryMapOutput.java:shuffle(100)) - Read 200 bytes from map-output for attempt_local1056486563_0003_m_000000_0
2015-02-12 16:12:46,658 INFO reduce.MergeManagerImpl (MergeManagerImpl.java:closeInMemoryFile(313)) - closeInMemoryFile -> map-output of size: 200, inMemoryMapOutputs.size() -> 2, commitMemory -> 200, usedMemory ->400
2015-02-12 16:12:46,658 INFO reduce.EventFetcher (EventFetcher.java:run(76)) - EventFetcher is interrupted.. Returning
2015-02-12 16:12:46,659 INFO mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) - 2 / 2 copied.
2015-02-12 16:12:46,659 INFO reduce.MergeManagerImpl (MergeManagerImpl.java:finalMerge(673)) - finalMerge called with 2 in-memory map-outputs and 0 on-disk map-outputs
2015-02-12 16:12:46,661 INFO mapred.Merger (Merger.java:merge(606)) - Merging 2 sorted segments
2015-02-12 16:12:46,661 INFO mapred.Merger (Merger.java:merge(705)) - Down to the last merge-pass, with 2 segments left of total size: 382 bytes
2015-02-12 16:12:46,662 INFO reduce.MergeManagerImpl (MergeManagerImpl.java:finalMerge(750)) - Merged 2 segments, 400 bytes to disk to satisfy reduce memory limit
2015-02-12 16:12:46,663 INFO reduce.MergeManagerImpl (MergeManagerImpl.java:finalMerge(780)) - Merging 1 files, 402 bytes from disk
2015-02-12 16:12:46,663 INFO reduce.MergeManagerImpl (MergeManagerImpl.java:finalMerge(795)) - Merging 0 segments, 0 bytes from memory into reduce
2015-02-12 16:12:46,663 INFO mapred.Merger (Merger.java:merge(606)) - Merging 1 sorted segments
2015-02-12 16:12:46,663 INFO mapred.Merger (Merger.java:merge(705)) - Down to the last merge-pass, with 1 segments left of total size: 389 bytes
2015-02-12 16:12:46,664 INFO mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) - 2 / 2 copied.
2015-02-12 16:12:46,664 INFO hadoop.FlowReducer (FlowReducer.java:configure(78)) - cascading version: 2.6.3
2015-02-12 16:12:46,664 INFO hadoop.FlowReducer (FlowReducer.java:configure(79)) - child jvm opts: -Xmx200m
2015-02-12 16:12:46,672 INFO hadoop.FlowReducer (FlowReducer.java:configure(97)) - sourcing from: GroupBy(average-lhs+average-rhs)[by: average-lhs:[{1}:'char'] average-rhs:[{1}:'char']]
2015-02-12 16:12:46,672 INFO hadoop.FlowReducer (FlowReducer.java:configure(100)) - sinking to: Hfs["TextDelimited[['char', 'average']]"]["/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/test/output/hadoop2-mr1/assemblyhelpersplatform/mergeaverage"]
2015-02-12 16:12:46,681 INFO mapred.Task (Task.java:done(1001)) - Task:attempt_local1056486563_0003_r_000000_0 is done. And is in the process of committing
2015-02-12 16:12:46,682 INFO mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) - 2 / 2 copied.
2015-02-12 16:12:46,683 INFO mapred.Task (Task.java:commit(1162)) - Task attempt_local1056486563_0003_r_000000_0 is allowed to commit now
2015-02-12 16:12:46,683 INFO output.FileOutputCommitter (FileOutputCommitter.java:commitTask(439)) - Saved output of task 'attempt_local1056486563_0003_r_000000_0' to file:/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/test/output/hadoop2-mr1/assemblyhelpersplatform/mergeaverage/_temporary/0/task_local1056486563_0003_r_000000
2015-02-12 16:12:46,684 INFO mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) - reduce > reduce
2015-02-12 16:12:46,684 INFO mapred.Task (Task.java:sendDone(1121)) - Task 'attempt_local1056486563_0003_r_000000_0' done.
2015-02-12 16:12:46,684 INFO mapred.LocalJobRunner (LocalJobRunner.java:run(325)) - Finishing task: attempt_local1056486563_0003_r_000000_0
2015-02-12 16:12:46,685 INFO mapred.LocalJobRunner (LocalJobRunner.java:runTasks(456)) - reduce task executor complete.
2015-02-12 16:12:46,705 INFO util.Hadoop18TapUtil (Hadoop18TapUtil.java:cleanTempPath(219)) - deleting temp path /Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/test/output/hadoop2-mr1/assemblyhelpersplatform/mergeaverage/_temporary
2015-02-12 16:12:46,746 INFO mapred.FileInputFormat (FileInputFormat.java:listStatus(247)) - Total input paths to process : 1
2015-02-12 16:12:46,770 INFO mapred.FileInputFormat (FileInputFormat.java:listStatus(247)) - Total input paths to process : 1
2015-02-12 16:12:46,771 INFO cascading.PlatformTestCase (PlatformTestCase.java:tearDown(209)) - copying to local /Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/test/output//hadoop2-mr1/assemblyhelpersplatform/mergeaverage
2015-02-12 16:12:46,786 INFO util.HadoopUtil (HadoopUtil.java:findMainClass(336)) - using default application jar, may cause class not found exceptions on the cluster
2015-02-12 16:12:46,787 INFO planner.HadoopPlanner (HadoopPlanner.java:initialize(225)) - using application jar: /Users/soberoi/.gradle/caches/modules-2/files-2.1/cascading/cascading-hadoop2-mr1/2.6.3/9ae016e2b8d246b9c1de9168e715d9f1b971aa4a/cascading-hadoop2-mr1-2.6.3.jar
2015-02-12 16:12:46,792 INFO Configuration.deprecation (Configuration.java:warnOnceIfDeprecated(1022)) - mapred.used.genericoptionsparser is deprecated. Instead, use mapreduce.client.genericoptionsparser.used
2015-02-12 16:12:46,793 INFO Configuration.deprecation (Configuration.java:warnOnceIfDeprecated(1022)) - mapred.job.tracker is deprecated. Instead, use mapreduce.jobtracker.address
2015-02-12 16:12:46,794 INFO Configuration.deprecation (Configuration.java:warnOnceIfDeprecated(1022)) - mapred.output.compress is deprecated. Instead, use mapreduce.output.fileoutputformat.compress
2015-02-12 16:12:46,802 INFO flow.Flow (BaseFlow.java:logInfo(1378)) - [shape] starting
2015-02-12 16:12:46,802 INFO flow.Flow (BaseFlow.java:logInfo(1378)) - [shape] source: Hfs["TextLine[['offset', 'line']->[ALL]]"]["/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/classes/test/data/lower.txt"]
2015-02-12 16:12:46,802 INFO flow.Flow (BaseFlow.java:logInfo(1378)) - [shape] sink: Hfs["TextLine[['line']->['item', 'element']]"]["/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/test/output/hadoop2-mr1/assemblyhelpersplatform/renameall"]
2015-02-12 16:12:46,803 INFO flow.Flow (BaseFlow.java:logInfo(1378)) - [shape] parallel execution is enabled: false
2015-02-12 16:12:46,803 INFO flow.Flow (BaseFlow.java:logInfo(1378)) - [shape] starting jobs: 1
2015-02-12 16:12:46,803 INFO flow.Flow (BaseFlow.java:logInfo(1378)) - [shape] allocating threads: 1
2015-02-12 16:12:46,803 INFO flow.FlowStep (BaseFlowStep.java:logInfo(834)) - [shape] starting step: (1/1) ...helpersplatform/renameall
2015-02-12 16:12:46,804 INFO jvm.JvmMetrics (JvmMetrics.java:init(71)) - Cannot initialize JVM Metrics with processName=JobTracker, sessionId= - already initialized
2015-02-12 16:12:46,806 INFO jvm.JvmMetrics (JvmMetrics.java:init(71)) - Cannot initialize JVM Metrics with processName=JobTracker, sessionId= - already initialized
2015-02-12 16:12:46,829 INFO mapred.FileInputFormat (FileInputFormat.java:listStatus(247)) - Total input paths to process : 1
2015-02-12 16:12:46,846 INFO mapreduce.JobSubmitter (JobSubmitter.java:submitJobInternal(492)) - number of splits:1
2015-02-12 16:12:46,860 INFO mapreduce.JobSubmitter (JobSubmitter.java:printTokens(575)) - Submitting tokens for job: job_local2036438259_0004
2015-02-12 16:12:46,874 WARN conf.Configuration (Configuration.java:loadProperty(2510)) - file:/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/tmp/cascading/staging/soberoi2036438259/.staging/job_local2036438259_0004/job.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.retry.interval; Ignoring.
2015-02-12 16:12:46,875 WARN conf.Configuration (Configuration.java:loadProperty(2510)) - file:/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/tmp/cascading/staging/soberoi2036438259/.staging/job_local2036438259_0004/job.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.attempts; Ignoring.
2015-02-12 16:12:46,915 WARN conf.Configuration (Configuration.java:loadProperty(2510)) - file:/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/test/tmp/mapred/local/localRunner/soberoi/job_local2036438259_0004/job_local2036438259_0004.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.retry.interval; Ignoring.
2015-02-12 16:12:46,916 WARN conf.Configuration (Configuration.java:loadProperty(2510)) - file:/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/test/tmp/mapred/local/localRunner/soberoi/job_local2036438259_0004/job_local2036438259_0004.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.attempts; Ignoring.
2015-02-12 16:12:46,917 INFO mapreduce.Job (Job.java:submit(1299)) - The url to track the job: http://localhost:8080/
2015-02-12 16:12:46,917 INFO flow.FlowStep (BaseFlowStep.java:logInfo(834)) - [shape] submitted hadoop job: job_local2036438259_0004
2015-02-12 16:12:46,917 INFO mapred.LocalJobRunner (LocalJobRunner.java:createOutputCommitter(471)) - OutputCommitter set in config null
2015-02-12 16:12:46,917 INFO flow.FlowStep (BaseFlowStep.java:logInfo(834)) - [shape] tracking url: http://localhost:8080/
2015-02-12 16:12:46,917 INFO mapred.LocalJobRunner (LocalJobRunner.java:createOutputCommitter(489)) - OutputCommitter is org.apache.hadoop.mapred.FileOutputCommitter
2015-02-12 16:12:46,919 INFO mapred.LocalJobRunner (LocalJobRunner.java:runTasks(448)) - Waiting for map tasks
2015-02-12 16:12:46,919 INFO mapred.LocalJobRunner (LocalJobRunner.java:run(224)) - Starting task: attempt_local2036438259_0004_m_000000_0
2015-02-12 16:12:46,920 INFO util.ProcfsBasedProcessTree (ProcfsBasedProcessTree.java:isAvailable(181)) - ProcfsBasedProcessTree currently is supported only on Linux.
2015-02-12 16:12:46,921 INFO mapred.Task (Task.java:initialize(587)) - Using ResourceCalculatorProcessTree : null
2015-02-12 16:12:46,921 INFO io.MultiInputSplit (MultiInputSplit.java:readFields(161)) - current split input path: file:/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/classes/test/data/lower.txt
2015-02-12 16:12:46,922 INFO mapred.MapTask (MapTask.java:updateJobWithSplit(482)) - Processing split: cascading.tap.hadoop.io.MultiInputSplit@4a27dbfc
2015-02-12 16:12:46,922 INFO mapred.MapTask (MapTask.java:runOldMapper(436)) - numReduceTasks: 0
2015-02-12 16:12:46,929 INFO hadoop.FlowMapper (FlowMapper.java:configure(71)) - cascading version: 2.6.3
2015-02-12 16:12:46,929 INFO hadoop.FlowMapper (FlowMapper.java:configure(72)) - child jvm opts: -Xmx200m
2015-02-12 16:12:46,933 INFO hadoop.FlowMapper (FlowMapper.java:configure(87)) - sourcing from: Hfs["TextLine[['offset', 'line']->[ALL]]"]["/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/classes/test/data/lower.txt"]
2015-02-12 16:12:46,933 INFO hadoop.FlowMapper (FlowMapper.java:configure(90)) - sinking to: Hfs["TextLine[['line']->['item', 'element']]"]["/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/test/output/hadoop2-mr1/assemblyhelpersplatform/renameall"]
2015-02-12 16:12:46,936 INFO mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) -
2015-02-12 16:12:46,936 INFO mapred.Task (Task.java:done(1001)) - Task:attempt_local2036438259_0004_m_000000_0 is done. And is in the process of committing
2015-02-12 16:12:46,937 INFO mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) -
2015-02-12 16:12:46,937 INFO mapred.Task (Task.java:commit(1162)) - Task attempt_local2036438259_0004_m_000000_0 is allowed to commit now
2015-02-12 16:12:46,938 INFO output.FileOutputCommitter (FileOutputCommitter.java:commitTask(439)) - Saved output of task 'attempt_local2036438259_0004_m_000000_0' to file:/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/test/output/hadoop2-mr1/assemblyhelpersplatform/renameall/_temporary/0/task_local2036438259_0004_m_000000
2015-02-12 16:12:46,939 INFO mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) - file:/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/classes/test/data/lower.txt:0+19
2015-02-12 16:12:46,939 INFO mapred.Task (Task.java:sendDone(1121)) - Task 'attempt_local2036438259_0004_m_000000_0' done.
2015-02-12 16:12:46,939 INFO mapred.LocalJobRunner (LocalJobRunner.java:run(249)) - Finishing task: attempt_local2036438259_0004_m_000000_0
2015-02-12 16:12:46,939 INFO mapred.LocalJobRunner (LocalJobRunner.java:runTasks(456)) - map task executor complete.
2015-02-12 16:12:46,955 INFO util.Hadoop18TapUtil (Hadoop18TapUtil.java:cleanTempPath(219)) - deleting temp path /Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/test/output/hadoop2-mr1/assemblyhelpersplatform/renameall/_temporary
2015-02-12 16:12:46,977 INFO mapred.FileInputFormat (FileInputFormat.java:listStatus(247)) - Total input paths to process : 1
2015-02-12 16:12:46,978 INFO cascading.PlatformTestCase (PlatformTestCase.java:tearDown(209)) - copying to local /Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/test/output//hadoop2-mr1/assemblyhelpersplatform/renameall
2015-02-12 16:12:46,993 INFO util.HadoopUtil (HadoopUtil.java:findMainClass(336)) - using default application jar, may cause class not found exceptions on the cluster
2015-02-12 16:12:46,993 INFO planner.HadoopPlanner (HadoopPlanner.java:initialize(225)) - using application jar: /Users/soberoi/.gradle/caches/modules-2/files-2.1/cascading/cascading-hadoop2-mr1/2.6.3/9ae016e2b8d246b9c1de9168e715d9f1b971aa4a/cascading-hadoop2-mr1-2.6.3.jar
2015-02-12 16:12:47,005 INFO Configuration.deprecation (Configuration.java:warnOnceIfDeprecated(1022)) - mapred.used.genericoptionsparser is deprecated. Instead, use mapreduce.client.genericoptionsparser.used
2015-02-12 16:12:47,008 INFO Configuration.deprecation (Configuration.java:warnOnceIfDeprecated(1022)) - mapred.job.tracker is deprecated. Instead, use mapreduce.jobtracker.address
2015-02-12 16:12:47,018 INFO Configuration.deprecation (Configuration.java:warnOnceIfDeprecated(1022)) - mapred.output.compress is deprecated. Instead, use mapreduce.output.fileoutputformat.compress
2015-02-12 16:12:47,024 INFO flow.Flow (BaseFlow.java:logInfo(1378)) - [count] starting
2015-02-12 16:12:47,024 INFO flow.Flow (BaseFlow.java:logInfo(1378)) - [count] source: Hfs["TextDelimited[['num', 'char']]"]["/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/classes/test/data/lhs.txt"]
2015-02-12 16:12:47,025 INFO flow.Flow (BaseFlow.java:logInfo(1378)) - [count] sink: Hfs["TextDelimited[['count', 'count2']]"]["/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/test/output/hadoop2-mr1/assemblyhelpersplatform/countcount"]
2015-02-12 16:12:47,025 INFO flow.Flow (BaseFlow.java:logInfo(1378)) - [count] parallel execution is enabled: false
2015-02-12 16:12:47,025 INFO flow.Flow (BaseFlow.java:logInfo(1378)) - [count] starting jobs: 2
2015-02-12 16:12:47,025 INFO flow.Flow (BaseFlow.java:logInfo(1378)) - [count] allocating threads: 1
2015-02-12 16:12:47,025 INFO flow.FlowStep (BaseFlowStep.java:logInfo(834)) - [count] starting step: (1/2)
2015-02-12 16:12:47,026 INFO jvm.JvmMetrics (JvmMetrics.java:init(71)) - Cannot initialize JVM Metrics with processName=JobTracker, sessionId= - already initialized
2015-02-12 16:12:47,027 INFO jvm.JvmMetrics (JvmMetrics.java:init(71)) - Cannot initialize JVM Metrics with processName=JobTracker, sessionId= - already initialized
2015-02-12 16:12:47,051 INFO mapred.FileInputFormat (FileInputFormat.java:listStatus(247)) - Total input paths to process : 1
2015-02-12 16:12:47,069 INFO mapreduce.JobSubmitter (JobSubmitter.java:submitJobInternal(492)) - number of splits:1
2015-02-12 16:12:47,090 INFO mapreduce.JobSubmitter (JobSubmitter.java:printTokens(575)) - Submitting tokens for job: job_local204371883_0005
2015-02-12 16:12:47,102 WARN conf.Configuration (Configuration.java:loadProperty(2510)) - file:/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/tmp/cascading/staging/soberoi204371883/.staging/job_local204371883_0005/job.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.retry.interval; Ignoring.
2015-02-12 16:12:47,103 WARN conf.Configuration (Configuration.java:loadProperty(2510)) - file:/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/tmp/cascading/staging/soberoi204371883/.staging/job_local204371883_0005/job.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.attempts; Ignoring.
2015-02-12 16:12:47,138 WARN conf.Configuration (Configuration.java:loadProperty(2510)) - file:/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/test/tmp/mapred/local/localRunner/soberoi/job_local204371883_0005/job_local204371883_0005.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.retry.interval; Ignoring.
2015-02-12 16:12:47,139 WARN conf.Configuration (Configuration.java:loadProperty(2510)) - file:/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/test/tmp/mapred/local/localRunner/soberoi/job_local204371883_0005/job_local204371883_0005.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.attempts; Ignoring.
2015-02-12 16:12:47,139 INFO mapreduce.Job (Job.java:submit(1299)) - The url to track the job: http://localhost:8080/
2015-02-12 16:12:47,140 INFO mapred.LocalJobRunner (LocalJobRunner.java:createOutputCommitter(471)) - OutputCommitter set in config null
2015-02-12 16:12:47,140 INFO flow.FlowStep (BaseFlowStep.java:logInfo(834)) - [count] submitted hadoop job: job_local204371883_0005
2015-02-12 16:12:47,140 INFO mapred.LocalJobRunner (LocalJobRunner.java:createOutputCommitter(489)) - OutputCommitter is org.apache.hadoop.mapred.FileOutputCommitter
2015-02-12 16:12:47,140 INFO flow.FlowStep (BaseFlowStep.java:logInfo(834)) - [count] tracking url: http://localhost:8080/
2015-02-12 16:12:47,142 INFO mapred.LocalJobRunner (LocalJobRunner.java:runTasks(448)) - Waiting for map tasks
2015-02-12 16:12:47,142 INFO mapred.LocalJobRunner (LocalJobRunner.java:run(224)) - Starting task: attempt_local204371883_0005_m_000000_0
2015-02-12 16:12:47,144 INFO util.ProcfsBasedProcessTree (ProcfsBasedProcessTree.java:isAvailable(181)) - ProcfsBasedProcessTree currently is supported only on Linux.
2015-02-12 16:12:47,144 INFO mapred.Task (Task.java:initialize(587)) - Using ResourceCalculatorProcessTree : null
2015-02-12 16:12:47,145 INFO io.MultiInputSplit (MultiInputSplit.java:readFields(161)) - current split input path: file:/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/classes/test/data/lhs.txt
2015-02-12 16:12:47,145 INFO mapred.MapTask (MapTask.java:updateJobWithSplit(482)) - Processing split: cascading.tap.hadoop.io.MultiInputSplit@7c4c3dc2
2015-02-12 16:12:47,146 INFO mapred.MapTask (MapTask.java:runOldMapper(436)) - numReduceTasks: 1
2015-02-12 16:12:47,158 INFO mapred.MapTask (MapTask.java:setEquator(1202)) - (EQUATOR) 0 kvi 26214396(104857584)
2015-02-12 16:12:47,158 INFO mapred.MapTask (MapTask.java:init(995)) - mapreduce.task.io.sort.mb: 100
2015-02-12 16:12:47,158 INFO mapred.MapTask (MapTask.java:init(996)) - soft limit at 83886080
2015-02-12 16:12:47,158 INFO mapred.MapTask (MapTask.java:init(997)) - bufstart = 0; bufvoid = 104857600
2015-02-12 16:12:47,158 INFO mapred.MapTask (MapTask.java:init(998)) - kvstart = 26214396; length = 6553600
2015-02-12 16:12:47,159 INFO mapred.MapTask (MapTask.java:createSortingCollector(402)) - Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer
2015-02-12 16:12:47,159 INFO hadoop.FlowMapper (FlowMapper.java:configure(71)) - cascading version: 2.6.3
2015-02-12 16:12:47,159 INFO hadoop.FlowMapper (FlowMapper.java:configure(72)) - child jvm opts: -Xmx200m
2015-02-12 16:12:47,166 INFO hadoop.FlowMapper (FlowMapper.java:configure(87)) - sourcing from: Hfs["TextDelimited[['num', 'char']]"]["/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/classes/test/data/lhs.txt"]
2015-02-12 16:12:47,166 INFO hadoop.FlowMapper (FlowMapper.java:configure(90)) - sinking to: GroupBy(count)[by:[{1}:'char']]
2015-02-12 16:12:47,166 INFO assembly.AggregateBy (AggregateBy.java:prepare(267)) - using threshold value: 2
2015-02-12 16:12:47,167 INFO assembly.AggregateBy (AggregateBy.java:removeEldestEntry(327)) - flushed keys num times: 1, with threshold: 2
2015-02-12 16:12:47,167 INFO assembly.AggregateBy (AggregateBy.java:removeEldestEntry(328)) - mem on flush (mb), free: 88, total: 509, max: 672
2015-02-12 16:12:47,167 INFO assembly.AggregateBy (AggregateBy.java:removeEldestEntry(333)) - total mem is 75% of max mem, to better utilize unused memory consider increasing current LRU threshold with system property "cascading.aggregateby.threshold"
2015-02-12 16:12:47,168 INFO assembly.AggregateBy (AggregateBy.java:removeEldestEntry(327)) - flushed keys num times: 3, with threshold: 2
2015-02-12 16:12:47,168 INFO assembly.AggregateBy (AggregateBy.java:removeEldestEntry(328)) - mem on flush (mb), free: 88, total: 509, max: 672
2015-02-12 16:12:47,168 INFO assembly.AggregateBy (AggregateBy.java:removeEldestEntry(333)) - total mem is 75% of max mem, to better utilize unused memory consider increasing current LRU threshold with system property "cascading.aggregateby.threshold"
2015-02-12 16:12:47,168 INFO assembly.AggregateBy (AggregateBy.java:removeEldestEntry(327)) - flushed keys num times: 5, with threshold: 2
2015-02-12 16:12:47,168 INFO assembly.AggregateBy (AggregateBy.java:removeEldestEntry(328)) - mem on flush (mb), free: 88, total: 509, max: 672
2015-02-12 16:12:47,169 INFO assembly.AggregateBy (AggregateBy.java:removeEldestEntry(333)) - total mem is 75% of max mem, to better utilize unused memory consider increasing current LRU threshold with system property "cascading.aggregateby.threshold"
2015-02-12 16:12:47,169 INFO assembly.AggregateBy (AggregateBy.java:removeEldestEntry(327)) - flushed keys num times: 7, with threshold: 2
2015-02-12 16:12:47,169 INFO assembly.AggregateBy (AggregateBy.java:removeEldestEntry(328)) - mem on flush (mb), free: 88, total: 509, max: 672
2015-02-12 16:12:47,169 INFO assembly.AggregateBy (AggregateBy.java:removeEldestEntry(333)) - total mem is 75% of max mem, to better utilize unused memory consider increasing current LRU threshold with system property "cascading.aggregateby.threshold"
2015-02-12 16:12:47,170 INFO mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) -
2015-02-12 16:12:47,170 INFO mapred.MapTask (MapTask.java:flush(1457)) - Starting flush of map output
2015-02-12 16:12:47,170 INFO mapred.MapTask (MapTask.java:flush(1475)) - Spilling map output
2015-02-12 16:12:47,170 INFO mapred.MapTask (MapTask.java:flush(1476)) - bufstart = 0; bufend = 99; bufvoid = 104857600
2015-02-12 16:12:47,170 INFO mapred.MapTask (MapTask.java:flush(1478)) - kvstart = 26214396(104857584); kvend = 26214364(104857456); length = 33/6553600
2015-02-12 16:12:47,172 INFO mapred.MapTask (MapTask.java:sortAndSpill(1660)) - Finished spill 0
2015-02-12 16:12:47,174 INFO mapred.Task (Task.java:done(1001)) - Task:attempt_local204371883_0005_m_000000_0 is done. And is in the process of committing
2015-02-12 16:12:47,175 INFO mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) - file:/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/classes/test/data/lhs.txt:0+52
2015-02-12 16:12:47,175 INFO mapred.Task (Task.java:sendDone(1121)) - Task 'attempt_local204371883_0005_m_000000_0' done.
2015-02-12 16:12:47,175 INFO mapred.LocalJobRunner (LocalJobRunner.java:run(249)) - Finishing task: attempt_local204371883_0005_m_000000_0
2015-02-12 16:12:47,175 INFO mapred.LocalJobRunner (LocalJobRunner.java:runTasks(456)) - map task executor complete.
2015-02-12 16:12:47,176 INFO mapred.LocalJobRunner (LocalJobRunner.java:runTasks(448)) - Waiting for reduce tasks
2015-02-12 16:12:47,176 INFO mapred.LocalJobRunner (LocalJobRunner.java:run(302)) - Starting task: attempt_local204371883_0005_r_000000_0
2015-02-12 16:12:47,177 INFO util.ProcfsBasedProcessTree (ProcfsBasedProcessTree.java:isAvailable(181)) - ProcfsBasedProcessTree currently is supported only on Linux.
2015-02-12 16:12:47,177 INFO mapred.Task (Task.java:initialize(587)) - Using ResourceCalculatorProcessTree : null
2015-02-12 16:12:47,177 INFO mapred.ReduceTask (ReduceTask.java:run(362)) - Using ShuffleConsumerPlugin: org.apache.hadoop.mapreduce.task.reduce.Shuffle@5d7566df
2015-02-12 16:12:47,178 INFO reduce.MergeManagerImpl (MergeManagerImpl.java:<init>(195)) - MergerManager: memoryLimit=493250144, maxSingleShuffleLimit=123312536, mergeThreshold=325545120, ioSortFactor=10, memToMemMergeOutputsThreshold=10
2015-02-12 16:12:47,179 INFO reduce.EventFetcher (EventFetcher.java:run(61)) - attempt_local204371883_0005_r_000000_0 Thread started: EventFetcher for fetching Map Completion Events
2015-02-12 16:12:47,180 INFO reduce.LocalFetcher (LocalFetcher.java:copyMapOutput(141)) - localfetcher#4 about to shuffle output of map attempt_local204371883_0005_m_000000_0 decomp: 119 len: 123 to MEMORY
2015-02-12 16:12:47,180 INFO reduce.InMemoryMapOutput (InMemoryMapOutput.java:shuffle(100)) - Read 119 bytes from map-output for attempt_local204371883_0005_m_000000_0
2015-02-12 16:12:47,180 INFO reduce.MergeManagerImpl (MergeManagerImpl.java:closeInMemoryFile(313)) - closeInMemoryFile -> map-output of size: 119, inMemoryMapOutputs.size() -> 1, commitMemory -> 0, usedMemory ->119
2015-02-12 16:12:47,181 INFO reduce.EventFetcher (EventFetcher.java:run(76)) - EventFetcher is interrupted.. Returning
2015-02-12 16:12:47,181 INFO mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) - 1 / 1 copied.
2015-02-12 16:12:47,182 INFO reduce.MergeManagerImpl (MergeManagerImpl.java:finalMerge(673)) - finalMerge called with 1 in-memory map-outputs and 0 on-disk map-outputs
2015-02-12 16:12:47,183 INFO mapred.Merger (Merger.java:merge(606)) - Merging 1 sorted segments
2015-02-12 16:12:47,184 INFO mapred.Merger (Merger.java:merge(705)) - Down to the last merge-pass, with 1 segments left of total size: 110 bytes
2015-02-12 16:12:47,185 INFO reduce.MergeManagerImpl (MergeManagerImpl.java:finalMerge(750)) - Merged 1 segments, 119 bytes to disk to satisfy reduce memory limit
2015-02-12 16:12:47,185 INFO reduce.MergeManagerImpl (MergeManagerImpl.java:finalMerge(780)) - Merging 1 files, 123 bytes from disk
2015-02-12 16:12:47,185 INFO reduce.MergeManagerImpl (MergeManagerImpl.java:finalMerge(795)) - Merging 0 segments, 0 bytes from memory into reduce
2015-02-12 16:12:47,185 INFO mapred.Merger (Merger.java:merge(606)) - Merging 1 sorted segments
2015-02-12 16:12:47,186 INFO mapred.Merger (Merger.java:merge(705)) - Down to the last merge-pass, with 1 segments left of total size: 110 bytes
2015-02-12 16:12:47,186 INFO mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) - 1 / 1 copied.
2015-02-12 16:12:47,186 INFO hadoop.FlowReducer (FlowReducer.java:configure(78)) - cascading version: 2.6.3
2015-02-12 16:12:47,187 INFO hadoop.FlowReducer (FlowReducer.java:configure(79)) - child jvm opts: -Xmx200m
2015-02-12 16:12:47,193 INFO hadoop.FlowReducer (FlowReducer.java:configure(97)) - sourcing from: GroupBy(count)[by:[{1}:'char']]
2015-02-12 16:12:47,193 INFO hadoop.FlowReducer (FlowReducer.java:configure(100)) - sinking to: TempHfs["SequenceFile[['count', 'count2']]"][7450648248/count/]
2015-02-12 16:12:47,207 INFO assembly.AggregateBy (AggregateBy.java:prepare(267)) - using threshold value: 2
2015-02-12 16:12:47,209 INFO assembly.AggregateBy (AggregateBy.java:removeEldestEntry(327)) - flushed keys num times: 1, with threshold: 2
2015-02-12 16:12:47,209 INFO assembly.AggregateBy (AggregateBy.java:removeEldestEntry(328)) - mem on flush (mb), free: 77, total: 509, max: 672
2015-02-12 16:12:47,209 INFO assembly.AggregateBy (AggregateBy.java:removeEldestEntry(333)) - total mem is 75% of max mem, to better utilize unused memory consider increasing current LRU threshold with system property "cascading.aggregateby.threshold"
2015-02-12 16:12:47,210 INFO mapred.Task (Task.java:done(1001)) - Task:attempt_local204371883_0005_r_000000_0 is done. And is in the process of committing
2015-02-12 16:12:47,211 INFO mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) - 1 / 1 copied.
2015-02-12 16:12:47,211 INFO mapred.Task (Task.java:commit(1162)) - Task attempt_local204371883_0005_r_000000_0 is allowed to commit now
2015-02-12 16:12:47,212 INFO output.FileOutputCommitter (FileOutputCommitter.java:commitTask(439)) - Saved output of task 'attempt_local204371883_0005_r_000000_0' to file:/tmp/hadoop-soberoi/7450648248_count_7A613910A9AE40EF949D3719A07AE1BF/_temporary/0/task_local204371883_0005_r_000000
2015-02-12 16:12:47,213 INFO mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) - reduce > reduce
2015-02-12 16:12:47,213 INFO mapred.Task (Task.java:sendDone(1121)) - Task 'attempt_local204371883_0005_r_000000_0' done.
2015-02-12 16:12:47,213 INFO mapred.LocalJobRunner (LocalJobRunner.java:run(325)) - Finishing task: attempt_local204371883_0005_r_000000_0
2015-02-12 16:12:47,213 INFO mapred.LocalJobRunner (LocalJobRunner.java:runTasks(456)) - reduce task executor complete.
2015-02-12 16:12:47,233 INFO flow.FlowStep (BaseFlowStep.java:logInfo(834)) - [count] starting step: (2/2) ...elpersplatform/countcount
2015-02-12 16:12:47,234 INFO jvm.JvmMetrics (JvmMetrics.java:init(71)) - Cannot initialize JVM Metrics with processName=JobTracker, sessionId= - already initialized
2015-02-12 16:12:47,235 INFO jvm.JvmMetrics (JvmMetrics.java:init(71)) - Cannot initialize JVM Metrics with processName=JobTracker, sessionId= - already initialized
2015-02-12 16:12:47,282 INFO mapred.FileInputFormat (FileInputFormat.java:listStatus(247)) - Total input paths to process : 1
2015-02-12 16:12:47,301 INFO mapreduce.JobSubmitter (JobSubmitter.java:submitJobInternal(492)) - number of splits:1
2015-02-12 16:12:47,311 INFO Configuration.deprecation (Configuration.java:warnOnceIfDeprecated(1022)) - mapred.used.genericoptionsparser is deprecated. Instead, use mapreduce.client.genericoptionsparser.used
2015-02-12 16:12:47,311 INFO Configuration.deprecation (Configuration.java:warnOnceIfDeprecated(1022)) - mapred.job.tracker is deprecated. Instead, use mapreduce.jobtracker.address
2015-02-12 16:12:47,315 INFO mapreduce.JobSubmitter (JobSubmitter.java:printTokens(575)) - Submitting tokens for job: job_local1176816738_0006
2015-02-12 16:12:47,329 WARN conf.Configuration (Configuration.java:loadProperty(2510)) - file:/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/tmp/cascading/staging/soberoi1176816738/.staging/job_local1176816738_0006/job.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.retry.interval; Ignoring.
2015-02-12 16:12:47,330 WARN conf.Configuration (Configuration.java:loadProperty(2510)) - file:/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/tmp/cascading/staging/soberoi1176816738/.staging/job_local1176816738_0006/job.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.attempts; Ignoring.
2015-02-12 16:12:47,362 WARN conf.Configuration (Configuration.java:loadProperty(2510)) - file:/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/test/tmp/mapred/local/localRunner/soberoi/job_local1176816738_0006/job_local1176816738_0006.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.retry.interval; Ignoring.
2015-02-12 16:12:47,363 WARN conf.Configuration (Configuration.java:loadProperty(2510)) - file:/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/test/tmp/mapred/local/localRunner/soberoi/job_local1176816738_0006/job_local1176816738_0006.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.attempts; Ignoring.
2015-02-12 16:12:47,364 INFO mapreduce.Job (Job.java:submit(1299)) - The url to track the job: http://localhost:8080/
2015-02-12 16:12:47,364 INFO mapred.LocalJobRunner (LocalJobRunner.java:createOutputCommitter(471)) - OutputCommitter set in config null
2015-02-12 16:12:47,364 INFO mapred.LocalJobRunner (LocalJobRunner.java:createOutputCommitter(489)) - OutputCommitter is org.apache.hadoop.mapred.FileOutputCommitter
2015-02-12 16:12:47,364 INFO flow.FlowStep (BaseFlowStep.java:logInfo(834)) - [count] submitted hadoop job: job_local1176816738_0006
2015-02-12 16:12:47,365 INFO flow.FlowStep (BaseFlowStep.java:logInfo(834)) - [count] tracking url: http://localhost:8080/
2015-02-12 16:12:47,367 INFO mapred.LocalJobRunner (LocalJobRunner.java:runTasks(448)) - Waiting for map tasks
2015-02-12 16:12:47,367 INFO mapred.LocalJobRunner (LocalJobRunner.java:run(224)) - Starting task: attempt_local1176816738_0006_m_000000_0
2015-02-12 16:12:47,368 INFO util.ProcfsBasedProcessTree (ProcfsBasedProcessTree.java:isAvailable(181)) - ProcfsBasedProcessTree currently is supported only on Linux.
2015-02-12 16:12:47,368 INFO mapred.Task (Task.java:initialize(587)) - Using ResourceCalculatorProcessTree : null
2015-02-12 16:12:47,370 INFO io.MultiInputSplit (MultiInputSplit.java:readFields(161)) - current split input path: file:/tmp/hadoop-soberoi/7450648248_count_7A613910A9AE40EF949D3719A07AE1BF/part-00000
2015-02-12 16:12:47,370 INFO mapred.MapTask (MapTask.java:updateJobWithSplit(482)) - Processing split: cascading.tap.hadoop.io.MultiInputSplit@346fadf4
2015-02-12 16:12:47,376 INFO mapred.MapTask (MapTask.java:runOldMapper(436)) - numReduceTasks: 1
2015-02-12 16:12:47,390 INFO mapred.MapTask (MapTask.java:setEquator(1202)) - (EQUATOR) 0 kvi 26214396(104857584)
2015-02-12 16:12:47,390 INFO mapred.MapTask (MapTask.java:init(995)) - mapreduce.task.io.sort.mb: 100
2015-02-12 16:12:47,390 INFO mapred.MapTask (MapTask.java:init(996)) - soft limit at 83886080
2015-02-12 16:12:47,390 INFO mapred.MapTask (MapTask.java:init(997)) - bufstart = 0; bufvoid = 104857600
2015-02-12 16:12:47,390 INFO mapred.MapTask (MapTask.java:init(998)) - kvstart = 26214396; length = 6553600
2015-02-12 16:12:47,391 INFO mapred.MapTask (MapTask.java:createSortingCollector(402)) - Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer
2015-02-12 16:12:47,391 INFO hadoop.FlowMapper (FlowMapper.java:configure(71)) - cascading version: 2.6.3
2015-02-12 16:12:47,391 INFO hadoop.FlowMapper (FlowMapper.java:configure(72)) - child jvm opts: -Xmx200m
2015-02-12 16:12:47,396 INFO hadoop.FlowMapper (FlowMapper.java:configure(87)) - sourcing from: TempHfs["SequenceFile[['count', 'count2']]"][7450648248/count/]
2015-02-12 16:12:47,397 INFO hadoop.FlowMapper (FlowMapper.java:configure(90)) - sinking to: GroupBy(count)[by:[{1}:'count']]
2015-02-12 16:12:47,398 INFO mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) -
2015-02-12 16:12:47,398 INFO mapred.MapTask (MapTask.java:flush(1457)) - Starting flush of map output
2015-02-12 16:12:47,398 INFO mapred.MapTask (MapTask.java:flush(1475)) - Spilling map output
2015-02-12 16:12:47,398 INFO mapred.MapTask (MapTask.java:flush(1476)) - bufstart = 0; bufend = 21; bufvoid = 104857600
2015-02-12 16:12:47,398 INFO mapred.MapTask (MapTask.java:flush(1478)) - kvstart = 26214396(104857584); kvend = 26214388(104857552); length = 9/6553600
2015-02-12 16:12:47,400 INFO mapred.MapTask (MapTask.java:sortAndSpill(1660)) - Finished spill 0
2015-02-12 16:12:47,402 INFO mapred.Task (Task.java:done(1001)) - Task:attempt_local1176816738_0006_m_000000_0 is done. And is in the process of committing
2015-02-12 16:12:47,403 INFO mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) - file:/tmp/hadoop-soberoi/7450648248_count_7A613910A9AE40EF949D3719A07AE1BF/part-00000:0+112
2015-02-12 16:12:47,403 INFO mapred.Task (Task.java:sendDone(1121)) - Task 'attempt_local1176816738_0006_m_000000_0' done.
2015-02-12 16:12:47,404 INFO mapred.LocalJobRunner (LocalJobRunner.java:run(249)) - Finishing task: attempt_local1176816738_0006_m_000000_0
2015-02-12 16:12:47,404 INFO mapred.LocalJobRunner (LocalJobRunner.java:runTasks(456)) - map task executor complete.
2015-02-12 16:12:47,404 INFO mapred.LocalJobRunner (LocalJobRunner.java:runTasks(448)) - Waiting for reduce tasks
2015-02-12 16:12:47,404 INFO mapred.LocalJobRunner (LocalJobRunner.java:run(302)) - Starting task: attempt_local1176816738_0006_r_000000_0
2015-02-12 16:12:47,406 INFO util.ProcfsBasedProcessTree (ProcfsBasedProcessTree.java:isAvailable(181)) - ProcfsBasedProcessTree currently is supported only on Linux.
2015-02-12 16:12:47,406 INFO mapred.Task (Task.java:initialize(587)) - Using ResourceCalculatorProcessTree : null
2015-02-12 16:12:47,406 INFO mapred.ReduceTask (ReduceTask.java:run(362)) - Using ShuffleConsumerPlugin: org.apache.hadoop.mapreduce.task.reduce.Shuffle@4fe087e7
2015-02-12 16:12:47,407 INFO reduce.MergeManagerImpl (MergeManagerImpl.java:<init>(195)) - MergerManager: memoryLimit=493250144, maxSingleShuffleLimit=123312536, mergeThreshold=325545120, ioSortFactor=10, memToMemMergeOutputsThreshold=10
2015-02-12 16:12:47,407 INFO reduce.EventFetcher (EventFetcher.java:run(61)) - attempt_local1176816738_0006_r_000000_0 Thread started: EventFetcher for fetching Map Completion Events
2015-02-12 16:12:47,409 INFO reduce.LocalFetcher (LocalFetcher.java:copyMapOutput(141)) - localfetcher#5 about to shuffle output of map attempt_local1176816738_0006_m_000000_0 decomp: 29 len: 33 to MEMORY
2015-02-12 16:12:47,409 INFO reduce.InMemoryMapOutput (InMemoryMapOutput.java:shuffle(100)) - Read 29 bytes from map-output for attempt_local1176816738_0006_m_000000_0
2015-02-12 16:12:47,409 INFO reduce.MergeManagerImpl (MergeManagerImpl.java:closeInMemoryFile(313)) - closeInMemoryFile -> map-output of size: 29, inMemoryMapOutputs.size() -> 1, commitMemory -> 0, usedMemory ->29
2015-02-12 16:12:47,410 INFO reduce.EventFetcher (EventFetcher.java:run(76)) - EventFetcher is interrupted.. Returning
2015-02-12 16:12:47,411 INFO mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) - 1 / 1 copied.
2015-02-12 16:12:47,411 INFO reduce.MergeManagerImpl (MergeManagerImpl.java:finalMerge(673)) - finalMerge called with 1 in-memory map-outputs and 0 on-disk map-outputs
2015-02-12 16:12:47,413 INFO mapred.Merger (Merger.java:merge(606)) - Merging 1 sorted segments
2015-02-12 16:12:47,413 INFO mapred.Merger (Merger.java:merge(705)) - Down to the last merge-pass, with 1 segments left of total size: 24 bytes
2015-02-12 16:12:47,415 INFO reduce.MergeManagerImpl (MergeManagerImpl.java:finalMerge(750)) - Merged 1 segments, 29 bytes to disk to satisfy reduce memory limit
2015-02-12 16:12:47,415 INFO reduce.MergeManagerImpl (MergeManagerImpl.java:finalMerge(780)) - Merging 1 files, 33 bytes from disk
2015-02-12 16:12:47,415 INFO reduce.MergeManagerImpl (MergeManagerImpl.java:finalMerge(795)) - Merging 0 segments, 0 bytes from memory into reduce
2015-02-12 16:12:47,415 INFO mapred.Merger (Merger.java:merge(606)) - Merging 1 sorted segments
2015-02-12 16:12:47,416 INFO mapred.Merger (Merger.java:merge(705)) - Down to the last merge-pass, with 1 segments left of total size: 24 bytes
2015-02-12 16:12:47,416 INFO mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) - 1 / 1 copied.
2015-02-12 16:12:47,416 INFO hadoop.FlowReducer (FlowReducer.java:configure(78)) - cascading version: 2.6.3
2015-02-12 16:12:47,417 INFO hadoop.FlowReducer (FlowReducer.java:configure(79)) - child jvm opts: -Xmx200m
2015-02-12 16:12:47,422 INFO hadoop.FlowReducer (FlowReducer.java:configure(97)) - sourcing from: GroupBy(count)[by:[{1}:'count']]
2015-02-12 16:12:47,422 INFO hadoop.FlowReducer (FlowReducer.java:configure(100)) - sinking to: Hfs["TextDelimited[['count', 'count2']]"]["/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/test/output/hadoop2-mr1/assemblyhelpersplatform/countcount"]
2015-02-12 16:12:47,431 INFO mapred.Task (Task.java:done(1001)) - Task:attempt_local1176816738_0006_r_000000_0 is done. And is in the process of committing
2015-02-12 16:12:47,432 INFO mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) - 1 / 1 copied.
2015-02-12 16:12:47,432 INFO mapred.Task (Task.java:commit(1162)) - Task attempt_local1176816738_0006_r_000000_0 is allowed to commit now
2015-02-12 16:12:47,433 INFO output.FileOutputCommitter (FileOutputCommitter.java:commitTask(439)) - Saved output of task 'attempt_local1176816738_0006_r_000000_0' to file:/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/test/output/hadoop2-mr1/assemblyhelpersplatform/countcount/_temporary/0/task_local1176816738_0006_r_000000
2015-02-12 16:12:47,434 INFO mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) - reduce > reduce
2015-02-12 16:12:47,434 INFO mapred.Task (Task.java:sendDone(1121)) - Task 'attempt_local1176816738_0006_r_000000_0' done.
2015-02-12 16:12:47,434 INFO mapred.LocalJobRunner (LocalJobRunner.java:run(325)) - Finishing task: attempt_local1176816738_0006_r_000000_0
2015-02-12 16:12:47,434 INFO mapred.LocalJobRunner (LocalJobRunner.java:runTasks(456)) - reduce task executor complete.
2015-02-12 16:12:47,454 INFO util.Hadoop18TapUtil (Hadoop18TapUtil.java:cleanTempPath(219)) - deleting temp path /Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/test/output/hadoop2-mr1/assemblyhelpersplatform/countcount/_temporary
2015-02-12 16:12:47,476 INFO mapred.FileInputFormat (FileInputFormat.java:listStatus(247)) - Total input paths to process : 1
2015-02-12 16:12:47,498 INFO mapred.FileInputFormat (FileInputFormat.java:listStatus(247)) - Total input paths to process : 1
2015-02-12 16:12:47,499 INFO cascading.PlatformTestCase (PlatformTestCase.java:tearDown(209)) - copying to local /Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/test/output//hadoop2-mr1/assemblyhelpersplatform/countcount
2015-02-12 16:12:47,515 INFO util.HadoopUtil (HadoopUtil.java:findMainClass(336)) - using default application jar, may cause class not found exceptions on the cluster
2015-02-12 16:12:47,516 INFO planner.HadoopPlanner (HadoopPlanner.java:initialize(225)) - using application jar: /Users/soberoi/.gradle/caches/modules-2/files-2.1/cascading/cascading-hadoop2-mr1/2.6.3/9ae016e2b8d246b9c1de9168e715d9f1b971aa4a/cascading-hadoop2-mr1-2.6.3.jar
2015-02-12 16:12:47,526 INFO Configuration.deprecation (Configuration.java:warnOnceIfDeprecated(1022)) - mapred.used.genericoptionsparser is deprecated. Instead, use mapreduce.client.genericoptionsparser.used
2015-02-12 16:12:47,529 INFO Configuration.deprecation (Configuration.java:warnOnceIfDeprecated(1022)) - mapred.job.tracker is deprecated. Instead, use mapreduce.jobtracker.address
2015-02-12 16:12:47,530 INFO Configuration.deprecation (Configuration.java:warnOnceIfDeprecated(1022)) - mapred.output.compress is deprecated. Instead, use mapreduce.output.fileoutputformat.compress
2015-02-12 16:12:47,537 INFO flow.Flow (BaseFlow.java:logInfo(1378)) - [count-lhs+count-rhs] starting
2015-02-12 16:12:47,538 INFO flow.Flow (BaseFlow.java:logInfo(1378)) - [count-lhs+count-rhs] source: Hfs["TextDelimited[['num', 'char']]"]["/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/classes/test/data/rhs.txt"]
2015-02-12 16:12:47,538 INFO flow.Flow (BaseFlow.java:logInfo(1378)) - [count-lhs+count-rhs] source: Hfs["TextDelimited[['num', 'char']]"]["/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/classes/test/data/lhs.txt"]
2015-02-12 16:12:47,538 INFO flow.Flow (BaseFlow.java:logInfo(1378)) - [count-lhs+count-rhs] sink: Hfs["TextDelimited[['char', 'count']]"]["/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/test/output/hadoop2-mr1/assemblyhelpersplatform/mergecount"]
2015-02-12 16:12:47,538 INFO flow.Flow (BaseFlow.java:logInfo(1378)) - [count-lhs+count-rhs] parallel execution is enabled: false
2015-02-12 16:12:47,538 INFO flow.Flow (BaseFlow.java:logInfo(1378)) - [count-lhs+count-rhs] starting jobs: 1
2015-02-12 16:12:47,539 INFO flow.Flow (BaseFlow.java:logInfo(1378)) - [count-lhs+count-rhs] allocating threads: 1
2015-02-12 16:12:47,539 INFO flow.FlowStep (BaseFlowStep.java:logInfo(834)) - [count-lhs+count-rhs] starting step: (1/1) ...elpersplatform/mergecount
2015-02-12 16:12:47,540 INFO jvm.JvmMetrics (JvmMetrics.java:init(71)) - Cannot initialize JVM Metrics with processName=JobTracker, sessionId= - already initialized
2015-02-12 16:12:47,540 INFO jvm.JvmMetrics (JvmMetrics.java:init(71)) - Cannot initialize JVM Metrics with processName=JobTracker, sessionId= - already initialized
2015-02-12 16:12:47,564 INFO mapred.FileInputFormat (FileInputFormat.java:listStatus(247)) - Total input paths to process : 1
2015-02-12 16:12:47,565 INFO mapred.FileInputFormat (FileInputFormat.java:listStatus(247)) - Total input paths to process : 1
2015-02-12 16:12:47,582 INFO mapreduce.JobSubmitter (JobSubmitter.java:submitJobInternal(492)) - number of splits:2
2015-02-12 16:12:47,595 INFO mapreduce.JobSubmitter (JobSubmitter.java:printTokens(575)) - Submitting tokens for job: job_local154647584_0007
2015-02-12 16:12:47,607 WARN conf.Configuration (Configuration.java:loadProperty(2510)) - file:/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/tmp/cascading/staging/soberoi154647584/.staging/job_local154647584_0007/job.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.retry.interval; Ignoring.
2015-02-12 16:12:47,607 WARN conf.Configuration (Configuration.java:loadProperty(2510)) - file:/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/tmp/cascading/staging/soberoi154647584/.staging/job_local154647584_0007/job.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.attempts; Ignoring.
2015-02-12 16:12:47,641 WARN conf.Configuration (Configuration.java:loadProperty(2510)) - file:/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/test/tmp/mapred/local/localRunner/soberoi/job_local154647584_0007/job_local154647584_0007.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.retry.interval; Ignoring.
2015-02-12 16:12:47,642 WARN conf.Configuration (Configuration.java:loadProperty(2510)) - file:/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/test/tmp/mapred/local/localRunner/soberoi/job_local154647584_0007/job_local154647584_0007.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.attempts; Ignoring.
2015-02-12 16:12:47,642 INFO mapreduce.Job (Job.java:submit(1299)) - The url to track the job: http://localhost:8080/
2015-02-12 16:12:47,642 INFO flow.FlowStep (BaseFlowStep.java:logInfo(834)) - [count-lhs+count-rhs] submitted hadoop job: job_local154647584_0007
2015-02-12 16:12:47,642 INFO mapred.LocalJobRunner (LocalJobRunner.java:createOutputCommitter(471)) - OutputCommitter set in config null
2015-02-12 16:12:47,642 INFO flow.FlowStep (BaseFlowStep.java:logInfo(834)) - [count-lhs+count-rhs] tracking url: http://localhost:8080/
2015-02-12 16:12:47,643 INFO mapred.LocalJobRunner (LocalJobRunner.java:createOutputCommitter(489)) - OutputCommitter is org.apache.hadoop.mapred.FileOutputCommitter
2015-02-12 16:12:47,645 INFO mapred.LocalJobRunner (LocalJobRunner.java:runTasks(448)) - Waiting for map tasks
2015-02-12 16:12:47,645 INFO mapred.LocalJobRunner (LocalJobRunner.java:run(224)) - Starting task: attempt_local154647584_0007_m_000000_0
2015-02-12 16:12:47,646 INFO util.ProcfsBasedProcessTree (ProcfsBasedProcessTree.java:isAvailable(181)) - ProcfsBasedProcessTree currently is supported only on Linux.
2015-02-12 16:12:47,646 INFO mapred.Task (Task.java:initialize(587)) - Using ResourceCalculatorProcessTree : null
2015-02-12 16:12:47,647 INFO io.MultiInputSplit (MultiInputSplit.java:readFields(161)) - current split input path: file:/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/classes/test/data/lhs.txt
2015-02-12 16:12:47,647 INFO mapred.MapTask (MapTask.java:updateJobWithSplit(482)) - Processing split: cascading.tap.hadoop.io.MultiInputSplit@27ea4128
2015-02-12 16:12:47,649 INFO mapred.MapTask (MapTask.java:runOldMapper(436)) - numReduceTasks: 1
2015-02-12 16:12:47,671 INFO mapred.MapTask (MapTask.java:setEquator(1202)) - (EQUATOR) 0 kvi 26214396(104857584)
2015-02-12 16:12:47,672 INFO mapred.MapTask (MapTask.java:init(995)) - mapreduce.task.io.sort.mb: 100
2015-02-12 16:12:47,673 INFO mapred.MapTask (MapTask.java:init(996)) - soft limit at 83886080
2015-02-12 16:12:47,673 INFO mapred.MapTask (MapTask.java:init(997)) - bufstart = 0; bufvoid = 104857600
2015-02-12 16:12:47,673 INFO mapred.MapTask (MapTask.java:init(998)) - kvstart = 26214396; length = 6553600
2015-02-12 16:12:47,674 INFO mapred.MapTask (MapTask.java:createSortingCollector(402)) - Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer
2015-02-12 16:12:47,674 INFO hadoop.FlowMapper (FlowMapper.java:configure(71)) - cascading version: 2.6.3
2015-02-12 16:12:47,674 INFO hadoop.FlowMapper (FlowMapper.java:configure(72)) - child jvm opts: -Xmx200m
2015-02-12 16:12:47,681 INFO hadoop.FlowMapper (FlowMapper.java:configure(87)) - sourcing from: Hfs["TextDelimited[['num', 'char']]"]["/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/classes/test/data/lhs.txt"]
2015-02-12 16:12:47,681 INFO hadoop.FlowMapper (FlowMapper.java:configure(90)) - sinking to: GroupBy(count-lhs+count-rhs)[by: count-lhs:[{1}:'char'] count-rhs:[{1}:'char']]
2015-02-12 16:12:47,681 INFO assembly.AggregateBy (AggregateBy.java:prepare(267)) - using threshold value: 2
2015-02-12 16:12:47,682 INFO assembly.AggregateBy (AggregateBy.java:removeEldestEntry(327)) - flushed keys num times: 1, with threshold: 2
2015-02-12 16:12:47,682 INFO assembly.AggregateBy (AggregateBy.java:removeEldestEntry(328)) - mem on flush (mb), free: 192, total: 502, max: 672
2015-02-12 16:12:47,682 INFO assembly.AggregateBy (AggregateBy.java:removeEldestEntry(333)) - total mem is 74% of max mem, to better utilize unused memory consider increasing current LRU threshold with system property "cascading.aggregateby.threshold"
2015-02-12 16:12:47,683 INFO assembly.AggregateBy (AggregateBy.java:removeEldestEntry(327)) - flushed keys num times: 3, with threshold: 2
2015-02-12 16:12:47,683 INFO assembly.AggregateBy (AggregateBy.java:removeEldestEntry(328)) - mem on flush (mb), free: 192, total: 502, max: 672
2015-02-12 16:12:47,683 INFO assembly.AggregateBy (AggregateBy.java:removeEldestEntry(333)) - total mem is 74% of max mem, to better utilize unused memory consider increasing current LRU threshold with system property "cascading.aggregateby.threshold"
2015-02-12 16:12:47,683 INFO assembly.AggregateBy (AggregateBy.java:removeEldestEntry(327)) - flushed keys num times: 5, with threshold: 2
2015-02-12 16:12:47,683 INFO assembly.AggregateBy (AggregateBy.java:removeEldestEntry(328)) - mem on flush (mb), free: 192, total: 502, max: 672
2015-02-12 16:12:47,683 INFO assembly.AggregateBy (AggregateBy.java:removeEldestEntry(333)) - total mem is 74% of max mem, to better utilize unused memory consider increasing current LRU threshold with system property "cascading.aggregateby.threshold"
2015-02-12 16:12:47,683 INFO assembly.AggregateBy (AggregateBy.java:removeEldestEntry(327)) - flushed keys num times: 7, with threshold: 2
2015-02-12 16:12:47,684 INFO assembly.AggregateBy (AggregateBy.java:removeEldestEntry(328)) - mem on flush (mb), free: 192, total: 502, max: 672
2015-02-12 16:12:47,684 INFO assembly.AggregateBy (AggregateBy.java:removeEldestEntry(333)) - total mem is 74% of max mem, to better utilize unused memory consider increasing current LRU threshold with system property "cascading.aggregateby.threshold"
2015-02-12 16:12:47,685 INFO mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) -
2015-02-12 16:12:47,685 INFO mapred.MapTask (MapTask.java:flush(1457)) - Starting flush of map output
2015-02-12 16:12:47,685 INFO mapred.MapTask (MapTask.java:flush(1475)) - Spilling map output
2015-02-12 16:12:47,685 INFO mapred.MapTask (MapTask.java:flush(1476)) - bufstart = 0; bufend = 99; bufvoid = 104857600
2015-02-12 16:12:47,685 INFO mapred.MapTask (MapTask.java:flush(1478)) - kvstart = 26214396(104857584); kvend = 26214364(104857456); length = 33/6553600
2015-02-12 16:12:47,687 INFO mapred.MapTask (MapTask.java:sortAndSpill(1660)) - Finished spill 0
2015-02-12 16:12:47,689 INFO mapred.Task (Task.java:done(1001)) - Task:attempt_local154647584_0007_m_000000_0 is done. And is in the process of committing
2015-02-12 16:12:47,690 INFO mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) - file:/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/classes/test/data/lhs.txt:0+52
2015-02-12 16:12:47,690 INFO mapred.Task (Task.java:sendDone(1121)) - Task 'attempt_local154647584_0007_m_000000_0' done.
2015-02-12 16:12:47,690 INFO mapred.LocalJobRunner (LocalJobRunner.java:run(249)) - Finishing task: attempt_local154647584_0007_m_000000_0
2015-02-12 16:12:47,690 INFO mapred.LocalJobRunner (LocalJobRunner.java:run(224)) - Starting task: attempt_local154647584_0007_m_000001_0
2015-02-12 16:12:47,691 INFO util.ProcfsBasedProcessTree (ProcfsBasedProcessTree.java:isAvailable(181)) - ProcfsBasedProcessTree currently is supported only on Linux.
2015-02-12 16:12:47,691 INFO mapred.Task (Task.java:initialize(587)) - Using ResourceCalculatorProcessTree : null
2015-02-12 16:12:47,692 INFO io.MultiInputSplit (MultiInputSplit.java:readFields(161)) - current split input path: file:/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/classes/test/data/rhs.txt
2015-02-12 16:12:47,692 INFO mapred.MapTask (MapTask.java:updateJobWithSplit(482)) - Processing split: cascading.tap.hadoop.io.MultiInputSplit@1d5ec199
2015-02-12 16:12:47,693 INFO mapred.MapTask (MapTask.java:runOldMapper(436)) - numReduceTasks: 1
2015-02-12 16:12:47,705 INFO mapred.MapTask (MapTask.java:setEquator(1202)) - (EQUATOR) 0 kvi 26214396(104857584)
2015-02-12 16:12:47,705 INFO mapred.MapTask (MapTask.java:init(995)) - mapreduce.task.io.sort.mb: 100
2015-02-12 16:12:47,705 INFO mapred.MapTask (MapTask.java:init(996)) - soft limit at 83886080
2015-02-12 16:12:47,705 INFO mapred.MapTask (MapTask.java:init(997)) - bufstart = 0; bufvoid = 104857600
2015-02-12 16:12:47,705 INFO mapred.MapTask (MapTask.java:init(998)) - kvstart = 26214396; length = 6553600
2015-02-12 16:12:47,706 INFO mapred.MapTask (MapTask.java:createSortingCollector(402)) - Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer
2015-02-12 16:12:47,706 INFO hadoop.FlowMapper (FlowMapper.java:configure(71)) - cascading version: 2.6.3
2015-02-12 16:12:47,706 INFO hadoop.FlowMapper (FlowMapper.java:configure(72)) - child jvm opts: -Xmx200m
2015-02-12 16:12:47,712 INFO hadoop.FlowMapper (FlowMapper.java:configure(87)) - sourcing from: Hfs["TextDelimited[['num', 'char']]"]["/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/classes/test/data/rhs.txt"]
2015-02-12 16:12:47,713 INFO hadoop.FlowMapper (FlowMapper.java:configure(90)) - sinking to: GroupBy(count-lhs+count-rhs)[by: count-lhs:[{1}:'char'] count-rhs:[{1}:'char']]
2015-02-12 16:12:47,713 INFO assembly.AggregateBy (AggregateBy.java:prepare(267)) - using threshold value: 2
2015-02-12 16:12:47,715 INFO assembly.AggregateBy (AggregateBy.java:removeEldestEntry(327)) - flushed keys num times: 1, with threshold: 2
2015-02-12 16:12:47,715 INFO assembly.AggregateBy (AggregateBy.java:removeEldestEntry(328)) - mem on flush (mb), free: 90, total: 502, max: 672
2015-02-12 16:12:47,715 INFO assembly.AggregateBy (AggregateBy.java:removeEldestEntry(333)) - total mem is 74% of max mem, to better utilize unused memory consider increasing current LRU threshold with system property "cascading.aggregateby.threshold"
2015-02-12 16:12:47,716 INFO assembly.AggregateBy (AggregateBy.java:removeEldestEntry(327)) - flushed keys num times: 3, with threshold: 2
2015-02-12 16:12:47,716 INFO assembly.AggregateBy (AggregateBy.java:removeEldestEntry(328)) - mem on flush (mb), free: 90, total: 502, max: 672
2015-02-12 16:12:47,716 INFO assembly.AggregateBy (AggregateBy.java:removeEldestEntry(333)) - total mem is 74% of max mem, to better utilize unused memory consider increasing current LRU threshold with system property "cascading.aggregateby.threshold"
2015-02-12 16:12:47,717 INFO assembly.AggregateBy (AggregateBy.java:removeEldestEntry(327)) - flushed keys num times: 5, with threshold: 2
2015-02-12 16:12:47,717 INFO assembly.AggregateBy (AggregateBy.java:removeEldestEntry(328)) - mem on flush (mb), free: 89, total: 502, max: 672
2015-02-12 16:12:47,717 INFO assembly.AggregateBy (AggregateBy.java:removeEldestEntry(333)) - total mem is 74% of max mem, to better utilize unused memory consider increasing current LRU threshold with system property "cascading.aggregateby.threshold"
2015-02-12 16:12:47,717 INFO assembly.AggregateBy (AggregateBy.java:removeEldestEntry(327)) - flushed keys num times: 7, with threshold: 2
2015-02-12 16:12:47,717 INFO assembly.AggregateBy (AggregateBy.java:removeEldestEntry(328)) - mem on flush (mb), free: 89, total: 502, max: 672
2015-02-12 16:12:47,717 INFO assembly.AggregateBy (AggregateBy.java:removeEldestEntry(333)) - total mem is 74% of max mem, to better utilize unused memory consider increasing current LRU threshold with system property "cascading.aggregateby.threshold"
2015-02-12 16:12:47,718 INFO mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) -
2015-02-12 16:12:47,718 INFO mapred.MapTask (MapTask.java:flush(1457)) - Starting flush of map output
2015-02-12 16:12:47,718 INFO mapred.MapTask (MapTask.java:flush(1475)) - Spilling map output
2015-02-12 16:12:47,718 INFO mapred.MapTask (MapTask.java:flush(1476)) - bufstart = 0; bufend = 99; bufvoid = 104857600
2015-02-12 16:12:47,718 INFO mapred.MapTask (MapTask.java:flush(1478)) - kvstart = 26214396(104857584); kvend = 26214364(104857456); length = 33/6553600
2015-02-12 16:12:47,720 INFO mapred.MapTask (MapTask.java:sortAndSpill(1660)) - Finished spill 0
2015-02-12 16:12:47,722 INFO mapred.Task (Task.java:done(1001)) - Task:attempt_local154647584_0007_m_000001_0 is done. And is in the process of committing
2015-02-12 16:12:47,723 INFO mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) - file:/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/classes/test/data/rhs.txt:0+51
2015-02-12 16:12:47,723 INFO mapred.Task (Task.java:sendDone(1121)) - Task 'attempt_local154647584_0007_m_000001_0' done.
2015-02-12 16:12:47,724 INFO mapred.LocalJobRunner (LocalJobRunner.java:run(249)) - Finishing task: attempt_local154647584_0007_m_000001_0
2015-02-12 16:12:47,724 INFO mapred.LocalJobRunner (LocalJobRunner.java:runTasks(456)) - map task executor complete.
2015-02-12 16:12:47,724 INFO mapred.LocalJobRunner (LocalJobRunner.java:runTasks(448)) - Waiting for reduce tasks
2015-02-12 16:12:47,724 INFO mapred.LocalJobRunner (LocalJobRunner.java:run(302)) - Starting task: attempt_local154647584_0007_r_000000_0
2015-02-12 16:12:47,726 INFO util.ProcfsBasedProcessTree (ProcfsBasedProcessTree.java:isAvailable(181)) - ProcfsBasedProcessTree currently is supported only on Linux.
2015-02-12 16:12:47,726 INFO mapred.Task (Task.java:initialize(587)) - Using ResourceCalculatorProcessTree : null
2015-02-12 16:12:47,726 INFO mapred.ReduceTask (ReduceTask.java:run(362)) - Using ShuffleConsumerPlugin: org.apache.hadoop.mapreduce.task.reduce.Shuffle@62aa34f5
2015-02-12 16:12:47,727 INFO reduce.MergeManagerImpl (MergeManagerImpl.java:<init>(195)) - MergerManager: memoryLimit=493250144, maxSingleShuffleLimit=123312536, mergeThreshold=325545120, ioSortFactor=10, memToMemMergeOutputsThreshold=10
2015-02-12 16:12:47,727 INFO reduce.EventFetcher (EventFetcher.java:run(61)) - attempt_local154647584_0007_r_000000_0 Thread started: EventFetcher for fetching Map Completion Events
2015-02-12 16:12:47,729 INFO reduce.LocalFetcher (LocalFetcher.java:copyMapOutput(141)) - localfetcher#6 about to shuffle output of map attempt_local154647584_0007_m_000001_0 decomp: 119 len: 123 to MEMORY
2015-02-12 16:12:47,729 INFO reduce.InMemoryMapOutput (InMemoryMapOutput.java:shuffle(100)) - Read 119 bytes from map-output for attempt_local154647584_0007_m_000001_0
2015-02-12 16:12:47,729 INFO reduce.MergeManagerImpl (MergeManagerImpl.java:closeInMemoryFile(313)) - closeInMemoryFile -> map-output of size: 119, inMemoryMapOutputs.size() -> 1, commitMemory -> 0, usedMemory ->119
2015-02-12 16:12:47,731 INFO reduce.LocalFetcher (LocalFetcher.java:copyMapOutput(141)) - localfetcher#6 about to shuffle output of map attempt_local154647584_0007_m_000000_0 decomp: 119 len: 123 to MEMORY
2015-02-12 16:12:47,732 INFO reduce.InMemoryMapOutput (InMemoryMapOutput.java:shuffle(100)) - Read 119 bytes from map-output for attempt_local154647584_0007_m_000000_0
2015-02-12 16:12:47,732 INFO reduce.MergeManagerImpl (MergeManagerImpl.java:closeInMemoryFile(313)) - closeInMemoryFile -> map-output of size: 119, inMemoryMapOutputs.size() -> 2, commitMemory -> 119, usedMemory ->238
2015-02-12 16:12:47,732 INFO reduce.EventFetcher (EventFetcher.java:run(76)) - EventFetcher is interrupted.. Returning
2015-02-12 16:12:47,733 INFO mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) - 2 / 2 copied.
2015-02-12 16:12:47,733 INFO reduce.MergeManagerImpl (MergeManagerImpl.java:finalMerge(673)) - finalMerge called with 2 in-memory map-outputs and 0 on-disk map-outputs
2015-02-12 16:12:47,735 INFO mapred.Merger (Merger.java:merge(606)) - Merging 2 sorted segments
2015-02-12 16:12:47,735 INFO mapred.Merger (Merger.java:merge(705)) - Down to the last merge-pass, with 2 segments left of total size: 220 bytes
2015-02-12 16:12:47,737 INFO reduce.MergeManagerImpl (MergeManagerImpl.java:finalMerge(750)) - Merged 2 segments, 238 bytes to disk to satisfy reduce memory limit
2015-02-12 16:12:47,737 INFO reduce.MergeManagerImpl (MergeManagerImpl.java:finalMerge(780)) - Merging 1 files, 240 bytes from disk
2015-02-12 16:12:47,738 INFO reduce.MergeManagerImpl (MergeManagerImpl.java:finalMerge(795)) - Merging 0 segments, 0 bytes from memory into reduce
2015-02-12 16:12:47,738 INFO mapred.Merger (Merger.java:merge(606)) - Merging 1 sorted segments
2015-02-12 16:12:47,738 INFO mapred.Merger (Merger.java:merge(705)) - Down to the last merge-pass, with 1 segments left of total size: 227 bytes
2015-02-12 16:12:47,739 INFO mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) - 2 / 2 copied.
2015-02-12 16:12:47,739 INFO hadoop.FlowReducer (FlowReducer.java:configure(78)) - cascading version: 2.6.3
2015-02-12 16:12:47,739 INFO hadoop.FlowReducer (FlowReducer.java:configure(79)) - child jvm opts: -Xmx200m
2015-02-12 16:12:47,746 INFO hadoop.FlowReducer (FlowReducer.java:configure(97)) - sourcing from: GroupBy(count-lhs+count-rhs)[by: count-lhs:[{1}:'char'] count-rhs:[{1}:'char']]
2015-02-12 16:12:47,746 INFO hadoop.FlowReducer (FlowReducer.java:configure(100)) - sinking to: Hfs["TextDelimited[['char', 'count']]"]["/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/test/output/hadoop2-mr1/assemblyhelpersplatform/mergecount"]
2015-02-12 16:12:47,755 INFO mapred.Task (Task.java:done(1001)) - Task:attempt_local154647584_0007_r_000000_0 is done. And is in the process of committing
2015-02-12 16:12:47,756 INFO mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) - 2 / 2 copied.
2015-02-12 16:12:47,756 INFO mapred.Task (Task.java:commit(1162)) - Task attempt_local154647584_0007_r_000000_0 is allowed to commit now
2015-02-12 16:12:47,757 INFO output.FileOutputCommitter (FileOutputCommitter.java:commitTask(439)) - Saved output of task 'attempt_local154647584_0007_r_000000_0' to file:/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/test/output/hadoop2-mr1/assemblyhelpersplatform/mergecount/_temporary/0/task_local154647584_0007_r_000000
2015-02-12 16:12:47,758 INFO mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) - reduce > reduce
2015-02-12 16:12:47,758 INFO mapred.Task (Task.java:sendDone(1121)) - Task 'attempt_local154647584_0007_r_000000_0' done.
2015-02-12 16:12:47,758 INFO mapred.LocalJobRunner (LocalJobRunner.java:run(325)) - Finishing task: attempt_local154647584_0007_r_000000_0
2015-02-12 16:12:47,758 INFO mapred.LocalJobRunner (LocalJobRunner.java:runTasks(456)) - reduce task executor complete.
2015-02-12 16:12:47,784 INFO util.Hadoop18TapUtil (Hadoop18TapUtil.java:cleanTempPath(219)) - deleting temp path /Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/test/output/hadoop2-mr1/assemblyhelpersplatform/mergecount/_temporary
2015-02-12 16:12:47,817 INFO mapred.FileInputFormat (FileInputFormat.java:listStatus(247)) - Total input paths to process : 1
2015-02-12 16:12:47,856 INFO mapred.FileInputFormat (FileInputFormat.java:listStatus(247)) - Total input paths to process : 1
2015-02-12 16:12:47,858 INFO cascading.PlatformTestCase (PlatformTestCase.java:tearDown(209)) - copying to local /Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/test/output//hadoop2-mr1/assemblyhelpersplatform/mergecount
2015-02-12 16:12:47,879 INFO util.HadoopUtil (HadoopUtil.java:findMainClass(336)) - using default application jar, may cause class not found exceptions on the cluster
2015-02-12 16:12:47,879 INFO planner.HadoopPlanner (HadoopPlanner.java:initialize(225)) - using application jar: /Users/soberoi/.gradle/caches/modules-2/files-2.1/cascading/cascading-hadoop2-mr1/2.6.3/9ae016e2b8d246b9c1de9168e715d9f1b971aa4a/cascading-hadoop2-mr1-2.6.3.jar
2015-02-12 16:12:47,886 INFO Configuration.deprecation (Configuration.java:warnOnceIfDeprecated(1022)) - mapred.used.genericoptionsparser is deprecated. Instead, use mapreduce.client.genericoptionsparser.used
2015-02-12 16:12:47,887 INFO Configuration.deprecation (Configuration.java:warnOnceIfDeprecated(1022)) - mapred.job.tracker is deprecated. Instead, use mapreduce.jobtracker.address
2015-02-12 16:12:47,888 INFO Configuration.deprecation (Configuration.java:warnOnceIfDeprecated(1022)) - mapred.output.compress is deprecated. Instead, use mapreduce.output.fileoutputformat.compress
2015-02-12 16:12:47,895 INFO flow.Flow (BaseFlow.java:logInfo(1378)) - [min] starting
2015-02-12 16:12:47,895 INFO flow.Flow (BaseFlow.java:logInfo(1378)) - [min] source: Hfs["TextDelimited[['num', 'char']]"]["/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/classes/test/data/lhs.txt"]
2015-02-12 16:12:47,896 INFO flow.Flow (BaseFlow.java:logInfo(1378)) - [min] sink: Hfs["TextDelimited[['char', 'min']]"]["/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/test/output/hadoop2-mr1/assemblyhelpersplatform/minbynullsafety"]
2015-02-12 16:12:47,896 INFO flow.Flow (BaseFlow.java:logInfo(1378)) - [min] parallel execution is enabled: false
2015-02-12 16:12:47,896 INFO flow.Flow (BaseFlow.java:logInfo(1378)) - [min] starting jobs: 1
2015-02-12 16:12:47,896 INFO flow.Flow (BaseFlow.java:logInfo(1378)) - [min] allocating threads: 1
2015-02-12 16:12:47,896 INFO flow.FlowStep (BaseFlowStep.java:logInfo(834)) - [min] starting step: (1/1) ...splatform/minbynullsafety
2015-02-12 16:12:47,897 INFO jvm.JvmMetrics (JvmMetrics.java:init(71)) - Cannot initialize JVM Metrics with processName=JobTracker, sessionId= - already initialized
2015-02-12 16:12:47,898 INFO jvm.JvmMetrics (JvmMetrics.java:init(71)) - Cannot initialize JVM Metrics with processName=JobTracker, sessionId= - already initialized
2015-02-12 16:12:47,918 INFO mapred.FileInputFormat (FileInputFormat.java:listStatus(247)) - Total input paths to process : 1
2015-02-12 16:12:47,937 INFO mapreduce.JobSubmitter (JobSubmitter.java:submitJobInternal(492)) - number of splits:1
2015-02-12 16:12:47,951 INFO mapreduce.JobSubmitter (JobSubmitter.java:printTokens(575)) - Submitting tokens for job: job_local521802607_0008
2015-02-12 16:12:47,964 WARN conf.Configuration (Configuration.java:loadProperty(2510)) - file:/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/tmp/cascading/staging/soberoi521802607/.staging/job_local521802607_0008/job.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.retry.interval; Ignoring.
2015-02-12 16:12:47,965 WARN conf.Configuration (Configuration.java:loadProperty(2510)) - file:/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/tmp/cascading/staging/soberoi521802607/.staging/job_local521802607_0008/job.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.attempts; Ignoring.
2015-02-12 16:12:47,996 WARN conf.Configuration (Configuration.java:loadProperty(2510)) - file:/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/test/tmp/mapred/local/localRunner/soberoi/job_local521802607_0008/job_local521802607_0008.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.retry.interval; Ignoring.
2015-02-12 16:12:47,997 WARN conf.Configuration (Configuration.java:loadProperty(2510)) - file:/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/test/tmp/mapred/local/localRunner/soberoi/job_local521802607_0008/job_local521802607_0008.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.attempts; Ignoring.
2015-02-12 16:12:47,998 INFO mapreduce.Job (Job.java:submit(1299)) - The url to track the job: http://localhost:8080/
2015-02-12 16:12:47,998 INFO mapred.LocalJobRunner (LocalJobRunner.java:createOutputCommitter(471)) - OutputCommitter set in config null
2015-02-12 16:12:47,998 INFO flow.FlowStep (BaseFlowStep.java:logInfo(834)) - [min] submitted hadoop job: job_local521802607_0008
2015-02-12 16:12:47,998 INFO mapred.LocalJobRunner (LocalJobRunner.java:createOutputCommitter(489)) - OutputCommitter is org.apache.hadoop.mapred.FileOutputCommitter
2015-02-12 16:12:47,998 INFO flow.FlowStep (BaseFlowStep.java:logInfo(834)) - [min] tracking url: http://localhost:8080/
2015-02-12 16:12:48,000 INFO mapred.LocalJobRunner (LocalJobRunner.java:runTasks(448)) - Waiting for map tasks
2015-02-12 16:12:48,001 INFO mapred.LocalJobRunner (LocalJobRunner.java:run(224)) - Starting task: attempt_local521802607_0008_m_000000_0
2015-02-12 16:12:48,002 INFO util.ProcfsBasedProcessTree (ProcfsBasedProcessTree.java:isAvailable(181)) - ProcfsBasedProcessTree currently is supported only on Linux.
2015-02-12 16:12:48,002 INFO mapred.Task (Task.java:initialize(587)) - Using ResourceCalculatorProcessTree : null
2015-02-12 16:12:48,003 INFO io.MultiInputSplit (MultiInputSplit.java:readFields(161)) - current split input path: file:/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/classes/test/data/lhs.txt
2015-02-12 16:12:48,003 INFO mapred.MapTask (MapTask.java:updateJobWithSplit(482)) - Processing split: cascading.tap.hadoop.io.MultiInputSplit@64cd6b49
2015-02-12 16:12:48,004 INFO mapred.MapTask (MapTask.java:runOldMapper(436)) - numReduceTasks: 1
2015-02-12 16:12:48,015 INFO mapred.MapTask (MapTask.java:setEquator(1202)) - (EQUATOR) 0 kvi 26214396(104857584)
2015-02-12 16:12:48,015 INFO mapred.MapTask (MapTask.java:init(995)) - mapreduce.task.io.sort.mb: 100
2015-02-12 16:12:48,015 INFO mapred.MapTask (MapTask.java:init(996)) - soft limit at 83886080
2015-02-12 16:12:48,016 INFO mapred.MapTask (MapTask.java:init(997)) - bufstart = 0; bufvoid = 104857600
2015-02-12 16:12:48,016 INFO mapred.MapTask (MapTask.java:init(998)) - kvstart = 26214396; length = 6553600
2015-02-12 16:12:48,016 INFO mapred.MapTask (MapTask.java:createSortingCollector(402)) - Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer
2015-02-12 16:12:48,016 INFO hadoop.FlowMapper (FlowMapper.java:configure(71)) - cascading version: 2.6.3
2015-02-12 16:12:48,017 INFO hadoop.FlowMapper (FlowMapper.java:configure(72)) - child jvm opts: -Xmx200m
2015-02-12 16:12:48,022 INFO hadoop.FlowMapper (FlowMapper.java:configure(87)) - sourcing from: Hfs["TextDelimited[['num', 'char']]"]["/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/classes/test/data/lhs.txt"]
2015-02-12 16:12:48,022 INFO hadoop.FlowMapper (FlowMapper.java:configure(90)) - sinking to: GroupBy(min)[by:[{1}:'char']]
2015-02-12 16:12:48,023 INFO assembly.AggregateBy (AggregateBy.java:prepare(267)) - using threshold value: 2
2015-02-12 16:12:48,024 INFO assembly.AggregateBy (AggregateBy.java:removeEldestEntry(327)) - flushed keys num times: 1, with threshold: 2
2015-02-12 16:12:48,024 INFO assembly.AggregateBy (AggregateBy.java:removeEldestEntry(328)) - mem on flush (mb), free: 158, total: 509, max: 672
2015-02-12 16:12:48,024 INFO assembly.AggregateBy (AggregateBy.java:removeEldestEntry(333)) - total mem is 75% of max mem, to better utilize unused memory consider increasing current LRU threshold with system property "cascading.aggregateby.threshold"
2015-02-12 16:12:48,024 INFO assembly.AggregateBy (AggregateBy.java:removeEldestEntry(327)) - flushed keys num times: 3, with threshold: 2
2015-02-12 16:12:48,025 INFO assembly.AggregateBy (AggregateBy.java:removeEldestEntry(328)) - mem on flush (mb), free: 158, total: 509, max: 672
2015-02-12 16:12:48,025 INFO assembly.AggregateBy (AggregateBy.java:removeEldestEntry(333)) - total mem is 75% of max mem, to better utilize unused memory consider increasing current LRU threshold with system property "cascading.aggregateby.threshold"
2015-02-12 16:12:48,025 INFO assembly.AggregateBy (AggregateBy.java:removeEldestEntry(327)) - flushed keys num times: 5, with threshold: 2
2015-02-12 16:12:48,025 INFO assembly.AggregateBy (AggregateBy.java:removeEldestEntry(328)) - mem on flush (mb), free: 158, total: 509, max: 672
2015-02-12 16:12:48,025 INFO assembly.AggregateBy (AggregateBy.java:removeEldestEntry(333)) - total mem is 75% of max mem, to better utilize unused memory consider increasing current LRU threshold with system property "cascading.aggregateby.threshold"
2015-02-12 16:12:48,026 INFO assembly.AggregateBy (AggregateBy.java:removeEldestEntry(327)) - flushed keys num times: 7, with threshold: 2
2015-02-12 16:12:48,026 INFO assembly.AggregateBy (AggregateBy.java:removeEldestEntry(328)) - mem on flush (mb), free: 158, total: 509, max: 672
2015-02-12 16:12:48,026 INFO assembly.AggregateBy (AggregateBy.java:removeEldestEntry(333)) - total mem is 75% of max mem, to better utilize unused memory consider increasing current LRU threshold with system property "cascading.aggregateby.threshold"
2015-02-12 16:12:48,027 INFO mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) -
2015-02-12 16:12:48,027 INFO mapred.MapTask (MapTask.java:flush(1457)) - Starting flush of map output
2015-02-12 16:12:48,027 INFO mapred.MapTask (MapTask.java:flush(1475)) - Spilling map output
2015-02-12 16:12:48,027 INFO mapred.MapTask (MapTask.java:flush(1476)) - bufstart = 0; bufend = 99; bufvoid = 104857600
2015-02-12 16:12:48,027 INFO mapred.MapTask (MapTask.java:flush(1478)) - kvstart = 26214396(104857584); kvend = 26214364(104857456); length = 33/6553600
2015-02-12 16:12:48,030 INFO mapred.MapTask (MapTask.java:sortAndSpill(1660)) - Finished spill 0
2015-02-12 16:12:48,032 INFO mapred.Task (Task.java:done(1001)) - Task:attempt_local521802607_0008_m_000000_0 is done. And is in the process of committing
2015-02-12 16:12:48,034 INFO mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) - file:/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/classes/test/data/lhs.txt:0+52
2015-02-12 16:12:48,034 INFO mapred.Task (Task.java:sendDone(1121)) - Task 'attempt_local521802607_0008_m_000000_0' done.
2015-02-12 16:12:48,034 INFO mapred.LocalJobRunner (LocalJobRunner.java:run(249)) - Finishing task: attempt_local521802607_0008_m_000000_0
2015-02-12 16:12:48,034 INFO mapred.LocalJobRunner (LocalJobRunner.java:runTasks(456)) - map task executor complete.
2015-02-12 16:12:48,035 INFO mapred.LocalJobRunner (LocalJobRunner.java:runTasks(448)) - Waiting for reduce tasks
2015-02-12 16:12:48,035 INFO mapred.LocalJobRunner (LocalJobRunner.java:run(302)) - Starting task: attempt_local521802607_0008_r_000000_0
2015-02-12 16:12:48,036 INFO util.ProcfsBasedProcessTree (ProcfsBasedProcessTree.java:isAvailable(181)) - ProcfsBasedProcessTree currently is supported only on Linux.
2015-02-12 16:12:48,036 INFO mapred.Task (Task.java:initialize(587)) - Using ResourceCalculatorProcessTree : null
2015-02-12 16:12:48,036 INFO mapred.ReduceTask (ReduceTask.java:run(362)) - Using ShuffleConsumerPlugin: org.apache.hadoop.mapreduce.task.reduce.Shuffle@263d40c
2015-02-12 16:12:48,037 INFO reduce.MergeManagerImpl (MergeManagerImpl.java:<init>(195)) - MergerManager: memoryLimit=493250144, maxSingleShuffleLimit=123312536, mergeThreshold=325545120, ioSortFactor=10, memToMemMergeOutputsThreshold=10
2015-02-12 16:12:48,037 INFO reduce.EventFetcher (EventFetcher.java:run(61)) - attempt_local521802607_0008_r_000000_0 Thread started: EventFetcher for fetching Map Completion Events
2015-02-12 16:12:48,040 INFO reduce.LocalFetcher (LocalFetcher.java:copyMapOutput(141)) - localfetcher#7 about to shuffle output of map attempt_local521802607_0008_m_000000_0 decomp: 119 len: 123 to MEMORY
2015-02-12 16:12:48,041 INFO reduce.InMemoryMapOutput (InMemoryMapOutput.java:shuffle(100)) - Read 119 bytes from map-output for attempt_local521802607_0008_m_000000_0
2015-02-12 16:12:48,041 INFO reduce.MergeManagerImpl (MergeManagerImpl.java:closeInMemoryFile(313)) - closeInMemoryFile -> map-output of size: 119, inMemoryMapOutputs.size() -> 1, commitMemory -> 0, usedMemory ->119
2015-02-12 16:12:48,042 INFO reduce.EventFetcher (EventFetcher.java:run(76)) - EventFetcher is interrupted.. Returning
2015-02-12 16:12:48,042 INFO mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) - 1 / 1 copied.
2015-02-12 16:12:48,042 INFO reduce.MergeManagerImpl (MergeManagerImpl.java:finalMerge(673)) - finalMerge called with 1 in-memory map-outputs and 0 on-disk map-outputs
2015-02-12 16:12:48,044 INFO mapred.Merger (Merger.java:merge(606)) - Merging 1 sorted segments
2015-02-12 16:12:48,044 INFO mapred.Merger (Merger.java:merge(705)) - Down to the last merge-pass, with 1 segments left of total size: 110 bytes
2015-02-12 16:12:48,046 INFO reduce.MergeManagerImpl (MergeManagerImpl.java:finalMerge(750)) - Merged 1 segments, 119 bytes to disk to satisfy reduce memory limit
2015-02-12 16:12:48,046 INFO reduce.MergeManagerImpl (MergeManagerImpl.java:finalMerge(780)) - Merging 1 files, 123 bytes from disk
2015-02-12 16:12:48,046 INFO reduce.MergeManagerImpl (MergeManagerImpl.java:finalMerge(795)) - Merging 0 segments, 0 bytes from memory into reduce
2015-02-12 16:12:48,046 INFO mapred.Merger (Merger.java:merge(606)) - Merging 1 sorted segments
2015-02-12 16:12:48,047 INFO mapred.Merger (Merger.java:merge(705)) - Down to the last merge-pass, with 1 segments left of total size: 110 bytes
2015-02-12 16:12:48,047 INFO mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) - 1 / 1 copied.
2015-02-12 16:12:48,047 INFO hadoop.FlowReducer (FlowReducer.java:configure(78)) - cascading version: 2.6.3
2015-02-12 16:12:48,047 INFO hadoop.FlowReducer (FlowReducer.java:configure(79)) - child jvm opts: -Xmx200m
2015-02-12 16:12:48,055 INFO hadoop.FlowReducer (FlowReducer.java:configure(97)) - sourcing from: GroupBy(min)[by:[{1}:'char']]
2015-02-12 16:12:48,055 INFO hadoop.FlowReducer (FlowReducer.java:configure(100)) - sinking to: Hfs["TextDelimited[['char', 'min']]"]["/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/test/output/hadoop2-mr1/assemblyhelpersplatform/minbynullsafety"]
2015-02-12 16:12:48,065 INFO mapred.Task (Task.java:done(1001)) - Task:attempt_local521802607_0008_r_000000_0 is done. And is in the process of committing
2015-02-12 16:12:48,066 INFO mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) - 1 / 1 copied.
2015-02-12 16:12:48,066 INFO mapred.Task (Task.java:commit(1162)) - Task attempt_local521802607_0008_r_000000_0 is allowed to commit now
2015-02-12 16:12:48,067 INFO output.FileOutputCommitter (FileOutputCommitter.java:commitTask(439)) - Saved output of task 'attempt_local521802607_0008_r_000000_0' to file:/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/test/output/hadoop2-mr1/assemblyhelpersplatform/minbynullsafety/_temporary/0/task_local521802607_0008_r_000000
2015-02-12 16:12:48,068 INFO mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) - reduce > reduce
2015-02-12 16:12:48,068 INFO mapred.Task (Task.java:sendDone(1121)) - Task 'attempt_local521802607_0008_r_000000_0' done.
2015-02-12 16:12:48,068 INFO mapred.LocalJobRunner (LocalJobRunner.java:run(325)) - Finishing task: attempt_local521802607_0008_r_000000_0
2015-02-12 16:12:48,068 INFO mapred.LocalJobRunner (LocalJobRunner.java:runTasks(456)) - reduce task executor complete.
2015-02-12 16:12:48,085 INFO util.Hadoop18TapUtil (Hadoop18TapUtil.java:cleanTempPath(219)) - deleting temp path /Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/test/output/hadoop2-mr1/assemblyhelpersplatform/minbynullsafety/_temporary
2015-02-12 16:12:48,107 INFO mapred.FileInputFormat (FileInputFormat.java:listStatus(247)) - Total input paths to process : 1
2015-02-12 16:12:48,139 INFO mapred.FileInputFormat (FileInputFormat.java:listStatus(247)) - Total input paths to process : 1
2015-02-12 16:12:48,141 INFO cascading.PlatformTestCase (PlatformTestCase.java:tearDown(209)) - copying to local /Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/test/output//hadoop2-mr1/assemblyhelpersplatform/minbynullsafety
2015-02-12 16:12:48,160 INFO util.HadoopUtil (HadoopUtil.java:findMainClass(336)) - using default application jar, may cause class not found exceptions on the cluster
2015-02-12 16:12:48,161 INFO planner.HadoopPlanner (HadoopPlanner.java:initialize(225)) - using application jar: /Users/soberoi/.gradle/caches/modules-2/files-2.1/cascading/cascading-hadoop2-mr1/2.6.3/9ae016e2b8d246b9c1de9168e715d9f1b971aa4a/cascading-hadoop2-mr1-2.6.3.jar
2015-02-12 16:12:48,166 INFO Configuration.deprecation (Configuration.java:warnOnceIfDeprecated(1022)) - mapred.used.genericoptionsparser is deprecated. Instead, use mapreduce.client.genericoptionsparser.used
2015-02-12 16:12:48,168 INFO Configuration.deprecation (Configuration.java:warnOnceIfDeprecated(1022)) - mapred.job.tracker is deprecated. Instead, use mapreduce.jobtracker.address
2015-02-12 16:12:48,168 INFO Configuration.deprecation (Configuration.java:warnOnceIfDeprecated(1022)) - mapred.output.compress is deprecated. Instead, use mapreduce.output.fileoutputformat.compress
2015-02-12 16:12:48,178 INFO flow.Flow (BaseFlow.java:logInfo(1378)) - [max] starting
2015-02-12 16:12:48,179 INFO flow.Flow (BaseFlow.java:logInfo(1378)) - [max] source: Hfs["TextDelimited[['num', 'char']]"]["/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/classes/test/data/lhs.txt"]
2015-02-12 16:12:48,179 INFO flow.Flow (BaseFlow.java:logInfo(1378)) - [max] sink: Hfs["TextDelimited[['num', 'max']]"]["/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/test/output/hadoop2-mr1/assemblyhelpersplatform/maxbystring"]
2015-02-12 16:12:48,179 INFO flow.Flow (BaseFlow.java:logInfo(1378)) - [max] parallel execution is enabled: false
2015-02-12 16:12:48,179 INFO flow.Flow (BaseFlow.java:logInfo(1378)) - [max] starting jobs: 1
2015-02-12 16:12:48,179 INFO flow.Flow (BaseFlow.java:logInfo(1378)) - [max] allocating threads: 1
2015-02-12 16:12:48,180 INFO flow.FlowStep (BaseFlowStep.java:logInfo(834)) - [max] starting step: (1/1) ...lpersplatform/maxbystring
2015-02-12 16:12:48,181 INFO jvm.JvmMetrics (JvmMetrics.java:init(71)) - Cannot initialize JVM Metrics with processName=JobTracker, sessionId= - already initialized
2015-02-12 16:12:48,182 INFO jvm.JvmMetrics (JvmMetrics.java:init(71)) - Cannot initialize JVM Metrics with processName=JobTracker, sessionId= - already initialized
2015-02-12 16:12:48,211 INFO mapred.FileInputFormat (FileInputFormat.java:listStatus(247)) - Total input paths to process : 1
2015-02-12 16:12:48,236 INFO mapreduce.JobSubmitter (JobSubmitter.java:submitJobInternal(492)) - number of splits:1
2015-02-12 16:12:48,253 INFO mapreduce.JobSubmitter (JobSubmitter.java:printTokens(575)) - Submitting tokens for job: job_local654358189_0009
2015-02-12 16:12:48,273 WARN conf.Configuration (Configuration.java:loadProperty(2510)) - file:/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/tmp/cascading/staging/soberoi654358189/.staging/job_local654358189_0009/job.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.retry.interval; Ignoring.
2015-02-12 16:12:48,274 WARN conf.Configuration (Configuration.java:loadProperty(2510)) - file:/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/tmp/cascading/staging/soberoi654358189/.staging/job_local654358189_0009/job.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.attempts; Ignoring.
2015-02-12 16:12:48,316 WARN conf.Configuration (Configuration.java:loadProperty(2510)) - file:/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/test/tmp/mapred/local/localRunner/soberoi/job_local654358189_0009/job_local654358189_0009.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.retry.interval; Ignoring.
2015-02-12 16:12:48,317 WARN conf.Configuration (Configuration.java:loadProperty(2510)) - file:/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/test/tmp/mapred/local/localRunner/soberoi/job_local654358189_0009/job_local654358189_0009.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.attempts; Ignoring.
2015-02-12 16:12:48,318 INFO mapreduce.Job (Job.java:submit(1299)) - The url to track the job: http://localhost:8080/
2015-02-12 16:12:48,318 INFO mapred.LocalJobRunner (LocalJobRunner.java:createOutputCommitter(471)) - OutputCommitter set in config null
2015-02-12 16:12:48,318 INFO flow.FlowStep (BaseFlowStep.java:logInfo(834)) - [max] submitted hadoop job: job_local654358189_0009
2015-02-12 16:12:48,318 INFO mapred.LocalJobRunner (LocalJobRunner.java:createOutputCommitter(489)) - OutputCommitter is org.apache.hadoop.mapred.FileOutputCommitter
2015-02-12 16:12:48,318 INFO flow.FlowStep (BaseFlowStep.java:logInfo(834)) - [max] tracking url: http://localhost:8080/
2015-02-12 16:12:48,321 INFO mapred.LocalJobRunner (LocalJobRunner.java:runTasks(448)) - Waiting for map tasks
2015-02-12 16:12:48,321 INFO mapred.LocalJobRunner (LocalJobRunner.java:run(224)) - Starting task: attempt_local654358189_0009_m_000000_0
2015-02-12 16:12:48,322 INFO util.ProcfsBasedProcessTree (ProcfsBasedProcessTree.java:isAvailable(181)) - ProcfsBasedProcessTree currently is supported only on Linux.
2015-02-12 16:12:48,322 INFO mapred.Task (Task.java:initialize(587)) - Using ResourceCalculatorProcessTree : null
2015-02-12 16:12:48,323 INFO io.MultiInputSplit (MultiInputSplit.java:readFields(161)) - current split input path: file:/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/classes/test/data/lhs.txt
2015-02-12 16:12:48,324 INFO mapred.MapTask (MapTask.java:updateJobWithSplit(482)) - Processing split: cascading.tap.hadoop.io.MultiInputSplit@4e5a67df
2015-02-12 16:12:48,325 INFO mapred.MapTask (MapTask.java:runOldMapper(436)) - numReduceTasks: 1
2015-02-12 16:12:48,341 INFO mapred.MapTask (MapTask.java:setEquator(1202)) - (EQUATOR) 0 kvi 26214396(104857584)
2015-02-12 16:12:48,341 INFO mapred.MapTask (MapTask.java:init(995)) - mapreduce.task.io.sort.mb: 100
2015-02-12 16:12:48,341 INFO mapred.MapTask (MapTask.java:init(996)) - soft limit at 83886080
2015-02-12 16:12:48,342 INFO mapred.MapTask (MapTask.java:init(997)) - bufstart = 0; bufvoid = 104857600
2015-02-12 16:12:48,342 INFO mapred.MapTask (MapTask.java:init(998)) - kvstart = 26214396; length = 6553600
2015-02-12 16:12:48,343 INFO mapred.MapTask (MapTask.java:createSortingCollector(402)) - Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer
2015-02-12 16:12:48,343 INFO hadoop.FlowMapper (FlowMapper.java:configure(71)) - cascading version: 2.6.3
2015-02-12 16:12:48,343 INFO hadoop.FlowMapper (FlowMapper.java:configure(72)) - child jvm opts: -Xmx200m
2015-02-12 16:12:48,348 INFO hadoop.FlowMapper (FlowMapper.java:configure(87)) - sourcing from: Hfs["TextDelimited[['num', 'char']]"]["/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/classes/test/data/lhs.txt"]
2015-02-12 16:12:48,348 INFO hadoop.FlowMapper (FlowMapper.java:configure(90)) - sinking to: GroupBy(max)[by:[{1}:'num']]
2015-02-12 16:12:48,348 INFO assembly.AggregateBy (AggregateBy.java:prepare(267)) - using threshold value: 2
2015-02-12 16:12:48,349 INFO assembly.AggregateBy (AggregateBy.java:removeEldestEntry(327)) - flushed keys num times: 1, with threshold: 2
2015-02-12 16:12:48,350 INFO assembly.AggregateBy (AggregateBy.java:removeEldestEntry(328)) - mem on flush (mb), free: 189, total: 499, max: 672
2015-02-12 16:12:48,350 INFO assembly.AggregateBy (AggregateBy.java:removeEldestEntry(333)) - total mem is 74% of max mem, to better utilize unused memory consider increasing current LRU threshold with system property "cascading.aggregateby.threshold"
2015-02-12 16:12:48,350 INFO assembly.AggregateBy (AggregateBy.java:removeEldestEntry(327)) - flushed keys num times: 3, with threshold: 2
2015-02-12 16:12:48,350 INFO assembly.AggregateBy (AggregateBy.java:removeEldestEntry(328)) - mem on flush (mb), free: 189, total: 499, max: 672
2015-02-12 16:12:48,350 INFO assembly.AggregateBy (AggregateBy.java:removeEldestEntry(333)) - total mem is 74% of max mem, to better utilize unused memory consider increasing current LRU threshold with system property "cascading.aggregateby.threshold"
2015-02-12 16:12:48,352 INFO mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) -
2015-02-12 16:12:48,352 INFO mapred.MapTask (MapTask.java:flush(1457)) - Starting flush of map output
2015-02-12 16:12:48,352 INFO mapred.MapTask (MapTask.java:flush(1475)) - Spilling map output
2015-02-12 16:12:48,352 INFO mapred.MapTask (MapTask.java:flush(1476)) - bufstart = 0; bufend = 75; bufvoid = 104857600
2015-02-12 16:12:48,352 INFO mapred.MapTask (MapTask.java:flush(1478)) - kvstart = 26214396(104857584); kvend = 26214380(104857520); length = 17/6553600
2015-02-12 16:12:48,354 INFO mapred.MapTask (MapTask.java:sortAndSpill(1660)) - Finished spill 0
2015-02-12 16:12:48,355 INFO mapred.Task (Task.java:done(1001)) - Task:attempt_local654358189_0009_m_000000_0 is done. And is in the process of committing
2015-02-12 16:12:48,356 INFO mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) - file:/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/classes/test/data/lhs.txt:0+52
2015-02-12 16:12:48,356 INFO mapred.Task (Task.java:sendDone(1121)) - Task 'attempt_local654358189_0009_m_000000_0' done.
2015-02-12 16:12:48,356 INFO mapred.LocalJobRunner (LocalJobRunner.java:run(249)) - Finishing task: attempt_local654358189_0009_m_000000_0
2015-02-12 16:12:48,356 INFO mapred.LocalJobRunner (LocalJobRunner.java:runTasks(456)) - map task executor complete.
2015-02-12 16:12:48,357 INFO mapred.LocalJobRunner (LocalJobRunner.java:runTasks(448)) - Waiting for reduce tasks
2015-02-12 16:12:48,357 INFO mapred.LocalJobRunner (LocalJobRunner.java:run(302)) - Starting task: attempt_local654358189_0009_r_000000_0
2015-02-12 16:12:48,358 INFO util.ProcfsBasedProcessTree (ProcfsBasedProcessTree.java:isAvailable(181)) - ProcfsBasedProcessTree currently is supported only on Linux.
2015-02-12 16:12:48,358 INFO mapred.Task (Task.java:initialize(587)) - Using ResourceCalculatorProcessTree : null
2015-02-12 16:12:48,358 INFO mapred.ReduceTask (ReduceTask.java:run(362)) - Using ShuffleConsumerPlugin: org.apache.hadoop.mapreduce.task.reduce.Shuffle@45b9d56f
2015-02-12 16:12:48,359 INFO reduce.MergeManagerImpl (MergeManagerImpl.java:<init>(195)) - MergerManager: memoryLimit=493250144, maxSingleShuffleLimit=123312536, mergeThreshold=325545120, ioSortFactor=10, memToMemMergeOutputsThreshold=10
2015-02-12 16:12:48,359 INFO reduce.EventFetcher (EventFetcher.java:run(61)) - attempt_local654358189_0009_r_000000_0 Thread started: EventFetcher for fetching Map Completion Events
2015-02-12 16:12:48,361 INFO reduce.LocalFetcher (LocalFetcher.java:copyMapOutput(141)) - localfetcher#8 about to shuffle output of map attempt_local654358189_0009_m_000000_0 decomp: 87 len: 91 to MEMORY
2015-02-12 16:12:48,361 INFO reduce.InMemoryMapOutput (InMemoryMapOutput.java:shuffle(100)) - Read 87 bytes from map-output for attempt_local654358189_0009_m_000000_0
2015-02-12 16:12:48,361 INFO reduce.MergeManagerImpl (MergeManagerImpl.java:closeInMemoryFile(313)) - closeInMemoryFile -> map-output of size: 87, inMemoryMapOutputs.size() -> 1, commitMemory -> 0, usedMemory ->87
2015-02-12 16:12:48,362 INFO reduce.EventFetcher (EventFetcher.java:run(76)) - EventFetcher is interrupted.. Returning
2015-02-12 16:12:48,362 INFO mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) - 1 / 1 copied.
2015-02-12 16:12:48,362 INFO reduce.MergeManagerImpl (MergeManagerImpl.java:finalMerge(673)) - finalMerge called with 1 in-memory map-outputs and 0 on-disk map-outputs
2015-02-12 16:12:48,364 INFO mapred.Merger (Merger.java:merge(606)) - Merging 1 sorted segments
2015-02-12 16:12:48,364 INFO mapred.Merger (Merger.java:merge(705)) - Down to the last merge-pass, with 1 segments left of total size: 78 bytes
2015-02-12 16:12:48,365 INFO reduce.MergeManagerImpl (MergeManagerImpl.java:finalMerge(750)) - Merged 1 segments, 87 bytes to disk to satisfy reduce memory limit
2015-02-12 16:12:48,365 INFO reduce.MergeManagerImpl (MergeManagerImpl.java:finalMerge(780)) - Merging 1 files, 91 bytes from disk
2015-02-12 16:12:48,366 INFO reduce.MergeManagerImpl (MergeManagerImpl.java:finalMerge(795)) - Merging 0 segments, 0 bytes from memory into reduce
2015-02-12 16:12:48,366 INFO mapred.Merger (Merger.java:merge(606)) - Merging 1 sorted segments
2015-02-12 16:12:48,366 INFO mapred.Merger (Merger.java:merge(705)) - Down to the last merge-pass, with 1 segments left of total size: 78 bytes
2015-02-12 16:12:48,366 INFO mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) - 1 / 1 copied.
2015-02-12 16:12:48,367 INFO hadoop.FlowReducer (FlowReducer.java:configure(78)) - cascading version: 2.6.3
2015-02-12 16:12:48,367 INFO hadoop.FlowReducer (FlowReducer.java:configure(79)) - child jvm opts: -Xmx200m
2015-02-12 16:12:48,373 INFO hadoop.FlowReducer (FlowReducer.java:configure(97)) - sourcing from: GroupBy(max)[by:[{1}:'num']]
2015-02-12 16:12:48,374 INFO hadoop.FlowReducer (FlowReducer.java:configure(100)) - sinking to: Hfs["TextDelimited[['num', 'max']]"]["/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/test/output/hadoop2-mr1/assemblyhelpersplatform/maxbystring"]
2015-02-12 16:12:48,385 INFO mapred.Task (Task.java:done(1001)) - Task:attempt_local654358189_0009_r_000000_0 is done. And is in the process of committing
2015-02-12 16:12:48,386 INFO mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) - 1 / 1 copied.
2015-02-12 16:12:48,386 INFO mapred.Task (Task.java:commit(1162)) - Task attempt_local654358189_0009_r_000000_0 is allowed to commit now
2015-02-12 16:12:48,387 INFO output.FileOutputCommitter (FileOutputCommitter.java:commitTask(439)) - Saved output of task 'attempt_local654358189_0009_r_000000_0' to file:/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/test/output/hadoop2-mr1/assemblyhelpersplatform/maxbystring/_temporary/0/task_local654358189_0009_r_000000
2015-02-12 16:12:48,388 INFO mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) - reduce > reduce
2015-02-12 16:12:48,388 INFO mapred.Task (Task.java:sendDone(1121)) - Task 'attempt_local654358189_0009_r_000000_0' done.
2015-02-12 16:12:48,388 INFO mapred.LocalJobRunner (LocalJobRunner.java:run(325)) - Finishing task: attempt_local654358189_0009_r_000000_0
2015-02-12 16:12:48,388 INFO mapred.LocalJobRunner (LocalJobRunner.java:runTasks(456)) - reduce task executor complete.
2015-02-12 16:12:48,408 INFO util.Hadoop18TapUtil (Hadoop18TapUtil.java:cleanTempPath(219)) - deleting temp path /Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/test/output/hadoop2-mr1/assemblyhelpersplatform/maxbystring/_temporary
2015-02-12 16:12:48,433 INFO mapred.FileInputFormat (FileInputFormat.java:listStatus(247)) - Total input paths to process : 1
2015-02-12 16:12:48,455 INFO mapred.FileInputFormat (FileInputFormat.java:listStatus(247)) - Total input paths to process : 1
2015-02-12 16:12:48,457 INFO cascading.PlatformTestCase (PlatformTestCase.java:tearDown(209)) - copying to local /Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/test/output//hadoop2-mr1/assemblyhelpersplatform/maxbystring
2015-02-12 16:12:48,474 INFO util.HadoopUtil (HadoopUtil.java:findMainClass(336)) - using default application jar, may cause class not found exceptions on the cluster
2015-02-12 16:12:48,475 INFO planner.HadoopPlanner (HadoopPlanner.java:initialize(225)) - using application jar: /Users/soberoi/.gradle/caches/modules-2/files-2.1/cascading/cascading-hadoop2-mr1/2.6.3/9ae016e2b8d246b9c1de9168e715d9f1b971aa4a/cascading-hadoop2-mr1-2.6.3.jar
2015-02-12 16:12:48,482 INFO Configuration.deprecation (Configuration.java:warnOnceIfDeprecated(1022)) - mapred.used.genericoptionsparser is deprecated. Instead, use mapreduce.client.genericoptionsparser.used
2015-02-12 16:12:48,484 INFO Configuration.deprecation (Configuration.java:warnOnceIfDeprecated(1022)) - mapred.job.tracker is deprecated. Instead, use mapreduce.jobtracker.address
2015-02-12 16:12:48,485 INFO Configuration.deprecation (Configuration.java:warnOnceIfDeprecated(1022)) - mapred.output.compress is deprecated. Instead, use mapreduce.output.fileoutputformat.compress
2015-02-12 16:12:48,496 INFO flow.Flow (BaseFlow.java:logInfo(1378)) - [first] starting
2015-02-12 16:12:48,497 INFO flow.Flow (BaseFlow.java:logInfo(1378)) - [first] source: Hfs["TextDelimited[['num', 'lower', 'upper']]"]["/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/classes/test/data/lhs+rhs-cross.txt"]
2015-02-12 16:12:48,497 INFO flow.Flow (BaseFlow.java:logInfo(1378)) - [first] sink: Hfs["TextDelimited[['num', 'lower', 'upper']]"]["/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/test/output/hadoop2-mr1/assemblyhelpersplatform/firstnfields"]
2015-02-12 16:12:48,497 INFO flow.Flow (BaseFlow.java:logInfo(1378)) - [first] parallel execution is enabled: false
2015-02-12 16:12:48,497 INFO flow.Flow (BaseFlow.java:logInfo(1378)) - [first] starting jobs: 1
2015-02-12 16:12:48,497 INFO flow.Flow (BaseFlow.java:logInfo(1378)) - [first] allocating threads: 1
2015-02-12 16:12:48,497 INFO flow.FlowStep (BaseFlowStep.java:logInfo(834)) - [first] starting step: (1/1) ...persplatform/firstnfields
2015-02-12 16:12:48,498 INFO jvm.JvmMetrics (JvmMetrics.java:init(71)) - Cannot initialize JVM Metrics with processName=JobTracker, sessionId= - already initialized
2015-02-12 16:12:48,498 INFO jvm.JvmMetrics (JvmMetrics.java:init(71)) - Cannot initialize JVM Metrics with processName=JobTracker, sessionId= - already initialized
2015-02-12 16:12:48,516 INFO mapred.FileInputFormat (FileInputFormat.java:listStatus(247)) - Total input paths to process : 1
2015-02-12 16:12:48,532 INFO mapreduce.JobSubmitter (JobSubmitter.java:submitJobInternal(492)) - number of splits:1
2015-02-12 16:12:48,543 INFO mapreduce.JobSubmitter (JobSubmitter.java:printTokens(575)) - Submitting tokens for job: job_local2080117784_0010
2015-02-12 16:12:48,553 WARN conf.Configuration (Configuration.java:loadProperty(2510)) - file:/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/tmp/cascading/staging/soberoi2080117784/.staging/job_local2080117784_0010/job.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.retry.interval; Ignoring.
2015-02-12 16:12:48,554 WARN conf.Configuration (Configuration.java:loadProperty(2510)) - file:/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/tmp/cascading/staging/soberoi2080117784/.staging/job_local2080117784_0010/job.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.attempts; Ignoring.
2015-02-12 16:12:48,589 WARN conf.Configuration (Configuration.java:loadProperty(2510)) - file:/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/test/tmp/mapred/local/localRunner/soberoi/job_local2080117784_0010/job_local2080117784_0010.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.retry.interval; Ignoring.
2015-02-12 16:12:48,590 WARN conf.Configuration (Configuration.java:loadProperty(2510)) - file:/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/test/tmp/mapred/local/localRunner/soberoi/job_local2080117784_0010/job_local2080117784_0010.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.attempts; Ignoring.
2015-02-12 16:12:48,591 INFO mapreduce.Job (Job.java:submit(1299)) - The url to track the job: http://localhost:8080/
2015-02-12 16:12:48,591 INFO mapred.LocalJobRunner (LocalJobRunner.java:createOutputCommitter(471)) - OutputCommitter set in config null
2015-02-12 16:12:48,591 INFO flow.FlowStep (BaseFlowStep.java:logInfo(834)) - [first] submitted hadoop job: job_local2080117784_0010
2015-02-12 16:12:48,591 INFO mapred.LocalJobRunner (LocalJobRunner.java:createOutputCommitter(489)) - OutputCommitter is org.apache.hadoop.mapred.FileOutputCommitter
2015-02-12 16:12:48,591 INFO flow.FlowStep (BaseFlowStep.java:logInfo(834)) - [first] tracking url: http://localhost:8080/
2015-02-12 16:12:48,593 INFO mapred.LocalJobRunner (LocalJobRunner.java:runTasks(448)) - Waiting for map tasks
2015-02-12 16:12:48,593 INFO mapred.LocalJobRunner (LocalJobRunner.java:run(224)) - Starting task: attempt_local2080117784_0010_m_000000_0
2015-02-12 16:12:48,594 INFO util.ProcfsBasedProcessTree (ProcfsBasedProcessTree.java:isAvailable(181)) - ProcfsBasedProcessTree currently is supported only on Linux.
2015-02-12 16:12:48,594 INFO mapred.Task (Task.java:initialize(587)) - Using ResourceCalculatorProcessTree : null
2015-02-12 16:12:48,595 INFO io.MultiInputSplit (MultiInputSplit.java:readFields(161)) - current split input path: file:/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/classes/test/data/lhs+rhs-cross.txt
2015-02-12 16:12:48,595 INFO mapred.MapTask (MapTask.java:updateJobWithSplit(482)) - Processing split: cascading.tap.hadoop.io.MultiInputSplit@676e582a
2015-02-12 16:12:48,595 INFO mapred.MapTask (MapTask.java:runOldMapper(436)) - numReduceTasks: 1
2015-02-12 16:12:48,611 INFO mapred.MapTask (MapTask.java:setEquator(1202)) - (EQUATOR) 0 kvi 26214396(104857584)
2015-02-12 16:12:48,611 INFO mapred.MapTask (MapTask.java:init(995)) - mapreduce.task.io.sort.mb: 100
2015-02-12 16:12:48,611 INFO mapred.MapTask (MapTask.java:init(996)) - soft limit at 83886080
2015-02-12 16:12:48,611 INFO mapred.MapTask (MapTask.java:init(997)) - bufstart = 0; bufvoid = 104857600
2015-02-12 16:12:48,611 INFO mapred.MapTask (MapTask.java:init(998)) - kvstart = 26214396; length = 6553600
2015-02-12 16:12:48,613 INFO mapred.MapTask (MapTask.java:createSortingCollector(402)) - Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer
2015-02-12 16:12:48,613 INFO hadoop.FlowMapper (FlowMapper.java:configure(71)) - cascading version: 2.6.3
2015-02-12 16:12:48,613 INFO hadoop.FlowMapper (FlowMapper.java:configure(72)) - child jvm opts: -Xmx200m
2015-02-12 16:12:48,618 INFO hadoop.FlowMapper (FlowMapper.java:configure(87)) - sourcing from: Hfs["TextDelimited[['num', 'lower', 'upper']]"]["/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/classes/test/data/lhs+rhs-cross.txt"]
2015-02-12 16:12:48,618 INFO hadoop.FlowMapper (FlowMapper.java:configure(90)) - sinking to: GroupBy(first)[by:[{1}:'num']]
2015-02-12 16:12:48,618 INFO assembly.AggregateBy (AggregateBy.java:prepare(267)) - using threshold value: 2
2015-02-12 16:12:48,620 INFO assembly.AggregateBy (AggregateBy.java:removeEldestEntry(327)) - flushed keys num times: 1, with threshold: 2
2015-02-12 16:12:48,620 INFO assembly.AggregateBy (AggregateBy.java:removeEldestEntry(328)) - mem on flush (mb), free: 200, total: 509, max: 672
2015-02-12 16:12:48,620 INFO assembly.AggregateBy (AggregateBy.java:removeEldestEntry(333)) - total mem is 75% of max mem, to better utilize unused memory consider increasing current LRU threshold with system property "cascading.aggregateby.threshold"
2015-02-12 16:12:48,621 INFO assembly.AggregateBy (AggregateBy.java:removeEldestEntry(327)) - flushed keys num times: 3, with threshold: 2
2015-02-12 16:12:48,621 INFO assembly.AggregateBy (AggregateBy.java:removeEldestEntry(328)) - mem on flush (mb), free: 199, total: 509, max: 672
2015-02-12 16:12:48,621 INFO assembly.AggregateBy (AggregateBy.java:removeEldestEntry(333)) - total mem is 75% of max mem, to better utilize unused memory consider increasing current LRU threshold with system property "cascading.aggregateby.threshold"
2015-02-12 16:12:48,622 INFO mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) -
2015-02-12 16:12:48,623 INFO mapred.MapTask (MapTask.java:flush(1457)) - Starting flush of map output
2015-02-12 16:12:48,623 INFO mapred.MapTask (MapTask.java:flush(1475)) - Spilling map output
2015-02-12 16:12:48,623 INFO mapred.MapTask (MapTask.java:flush(1476)) - bufstart = 0; bufend = 170; bufvoid = 104857600
2015-02-12 16:12:48,623 INFO mapred.MapTask (MapTask.java:flush(1478)) - kvstart = 26214396(104857584); kvend = 26214380(104857520); length = 17/6553600
2015-02-12 16:12:48,624 INFO mapred.MapTask (MapTask.java:sortAndSpill(1660)) - Finished spill 0
2015-02-12 16:12:48,626 INFO mapred.Task (Task.java:done(1001)) - Task:attempt_local2080117784_0010_m_000000_0 is done. And is in the process of committing
2015-02-12 16:12:48,627 INFO mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) - file:/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/classes/test/data/lhs+rhs-cross.txt:0+221
2015-02-12 16:12:48,627 INFO mapred.Task (Task.java:sendDone(1121)) - Task 'attempt_local2080117784_0010_m_000000_0' done.
2015-02-12 16:12:48,627 INFO mapred.LocalJobRunner (LocalJobRunner.java:run(249)) - Finishing task: attempt_local2080117784_0010_m_000000_0
2015-02-12 16:12:48,627 INFO mapred.LocalJobRunner (LocalJobRunner.java:runTasks(456)) - map task executor complete.
2015-02-12 16:12:48,628 INFO mapred.LocalJobRunner (LocalJobRunner.java:runTasks(448)) - Waiting for reduce tasks
2015-02-12 16:12:48,628 INFO mapred.LocalJobRunner (LocalJobRunner.java:run(302)) - Starting task: attempt_local2080117784_0010_r_000000_0
2015-02-12 16:12:48,629 INFO util.ProcfsBasedProcessTree (ProcfsBasedProcessTree.java:isAvailable(181)) - ProcfsBasedProcessTree currently is supported only on Linux.
2015-02-12 16:12:48,629 INFO mapred.Task (Task.java:initialize(587)) - Using ResourceCalculatorProcessTree : null
2015-02-12 16:12:48,629 INFO mapred.ReduceTask (ReduceTask.java:run(362)) - Using ShuffleConsumerPlugin: org.apache.hadoop.mapreduce.task.reduce.Shuffle@21c88db
2015-02-12 16:12:48,630 INFO reduce.MergeManagerImpl (MergeManagerImpl.java:<init>(195)) - MergerManager: memoryLimit=493250144, maxSingleShuffleLimit=123312536, mergeThreshold=325545120, ioSortFactor=10, memToMemMergeOutputsThreshold=10
2015-02-12 16:12:48,630 INFO reduce.EventFetcher (EventFetcher.java:run(61)) - attempt_local2080117784_0010_r_000000_0 Thread started: EventFetcher for fetching Map Completion Events
2015-02-12 16:12:48,631 INFO reduce.LocalFetcher (LocalFetcher.java:copyMapOutput(141)) - localfetcher#9 about to shuffle output of map attempt_local2080117784_0010_m_000000_0 decomp: 182 len: 186 to MEMORY
2015-02-12 16:12:48,631 INFO reduce.InMemoryMapOutput (InMemoryMapOutput.java:shuffle(100)) - Read 182 bytes from map-output for attempt_local2080117784_0010_m_000000_0
2015-02-12 16:12:48,631 INFO reduce.MergeManagerImpl (MergeManagerImpl.java:closeInMemoryFile(313)) - closeInMemoryFile -> map-output of size: 182, inMemoryMapOutputs.size() -> 1, commitMemory -> 0, usedMemory ->182
2015-02-12 16:12:48,632 INFO reduce.EventFetcher (EventFetcher.java:run(76)) - EventFetcher is interrupted.. Returning
2015-02-12 16:12:48,632 INFO mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) - 1 / 1 copied.
2015-02-12 16:12:48,633 INFO reduce.MergeManagerImpl (MergeManagerImpl.java:finalMerge(673)) - finalMerge called with 1 in-memory map-outputs and 0 on-disk map-outputs
2015-02-12 16:12:48,634 INFO mapred.Merger (Merger.java:merge(606)) - Merging 1 sorted segments
2015-02-12 16:12:48,634 INFO mapred.Merger (Merger.java:merge(705)) - Down to the last merge-pass, with 1 segments left of total size: 160 bytes
2015-02-12 16:12:48,636 INFO reduce.MergeManagerImpl (MergeManagerImpl.java:finalMerge(750)) - Merged 1 segments, 182 bytes to disk to satisfy reduce memory limit
2015-02-12 16:12:48,636 INFO reduce.MergeManagerImpl (MergeManagerImpl.java:finalMerge(780)) - Merging 1 files, 186 bytes from disk
2015-02-12 16:12:48,636 INFO reduce.MergeManagerImpl (MergeManagerImpl.java:finalMerge(795)) - Merging 0 segments, 0 bytes from memory into reduce
2015-02-12 16:12:48,636 INFO mapred.Merger (Merger.java:merge(606)) - Merging 1 sorted segments
2015-02-12 16:12:48,636 INFO mapred.Merger (Merger.java:merge(705)) - Down to the last merge-pass, with 1 segments left of total size: 160 bytes
2015-02-12 16:12:48,637 INFO mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) - 1 / 1 copied.
2015-02-12 16:12:48,637 INFO hadoop.FlowReducer (FlowReducer.java:configure(78)) - cascading version: 2.6.3
2015-02-12 16:12:48,637 INFO hadoop.FlowReducer (FlowReducer.java:configure(79)) - child jvm opts: -Xmx200m
2015-02-12 16:12:48,643 INFO hadoop.FlowReducer (FlowReducer.java:configure(97)) - sourcing from: GroupBy(first)[by:[{1}:'num']]
2015-02-12 16:12:48,643 INFO hadoop.FlowReducer (FlowReducer.java:configure(100)) - sinking to: Hfs["TextDelimited[['num', 'lower', 'upper']]"]["/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/test/output/hadoop2-mr1/assemblyhelpersplatform/firstnfields"]
2015-02-12 16:12:48,651 INFO mapred.Task (Task.java:done(1001)) - Task:attempt_local2080117784_0010_r_000000_0 is done. And is in the process of committing
2015-02-12 16:12:48,652 INFO mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) - 1 / 1 copied.
2015-02-12 16:12:48,652 INFO mapred.Task (Task.java:commit(1162)) - Task attempt_local2080117784_0010_r_000000_0 is allowed to commit now
2015-02-12 16:12:48,653 INFO output.FileOutputCommitter (FileOutputCommitter.java:commitTask(439)) - Saved output of task 'attempt_local2080117784_0010_r_000000_0' to file:/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/test/output/hadoop2-mr1/assemblyhelpersplatform/firstnfields/_temporary/0/task_local2080117784_0010_r_000000
2015-02-12 16:12:48,653 INFO mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) - reduce > reduce
2015-02-12 16:12:48,653 INFO mapred.Task (Task.java:sendDone(1121)) - Task 'attempt_local2080117784_0010_r_000000_0' done.
2015-02-12 16:12:48,654 INFO mapred.LocalJobRunner (LocalJobRunner.java:run(325)) - Finishing task: attempt_local2080117784_0010_r_000000_0
2015-02-12 16:12:48,654 INFO mapred.LocalJobRunner (LocalJobRunner.java:runTasks(456)) - reduce task executor complete.
2015-02-12 16:12:48,673 INFO util.Hadoop18TapUtil (Hadoop18TapUtil.java:cleanTempPath(219)) - deleting temp path /Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/test/output/hadoop2-mr1/assemblyhelpersplatform/firstnfields/_temporary
2015-02-12 16:12:48,695 INFO mapred.FileInputFormat (FileInputFormat.java:listStatus(247)) - Total input paths to process : 1
2015-02-12 16:12:48,696 INFO cascading.PlatformTestCase (PlatformTestCase.java:tearDown(209)) - copying to local /Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/test/output//hadoop2-mr1/assemblyhelpersplatform/firstnfields
2015-02-12 16:12:48,711 INFO util.HadoopUtil (HadoopUtil.java:findMainClass(336)) - using default application jar, may cause class not found exceptions on the cluster
2015-02-12 16:12:48,711 INFO planner.HadoopPlanner (HadoopPlanner.java:initialize(225)) - using application jar: /Users/soberoi/.gradle/caches/modules-2/files-2.1/cascading/cascading-hadoop2-mr1/2.6.3/9ae016e2b8d246b9c1de9168e715d9f1b971aa4a/cascading-hadoop2-mr1-2.6.3.jar
2015-02-12 16:12:48,716 INFO Configuration.deprecation (Configuration.java:warnOnceIfDeprecated(1022)) - mapred.used.genericoptionsparser is deprecated. Instead, use mapreduce.client.genericoptionsparser.used
2015-02-12 16:12:48,717 INFO Configuration.deprecation (Configuration.java:warnOnceIfDeprecated(1022)) - mapred.job.tracker is deprecated. Instead, use mapreduce.jobtracker.address
2015-02-12 16:12:48,717 INFO Configuration.deprecation (Configuration.java:warnOnceIfDeprecated(1022)) - mapred.output.compress is deprecated. Instead, use mapreduce.output.fileoutputformat.compress
2015-02-12 16:12:48,723 INFO flow.Flow (BaseFlow.java:logInfo(1378)) - [coerce] starting
2015-02-12 16:12:48,724 INFO flow.Flow (BaseFlow.java:logInfo(1378)) - [coerce] source: Hfs["TextLine[['offset', 'line']->[ALL]]"]["/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/classes/test/data/lower.txt"]
2015-02-12 16:12:48,724 INFO flow.Flow (BaseFlow.java:logInfo(1378)) - [coerce] sink: Hfs["TextLine[['line']->['num', 'char']]"]["/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/test/output/hadoop2-mr1/assemblyhelpersplatform/coercefields"]
2015-02-12 16:12:48,724 INFO flow.Flow (BaseFlow.java:logInfo(1378)) - [coerce] parallel execution is enabled: false
2015-02-12 16:12:48,724 INFO flow.Flow (BaseFlow.java:logInfo(1378)) - [coerce] starting jobs: 1
2015-02-12 16:12:48,724 INFO flow.Flow (BaseFlow.java:logInfo(1378)) - [coerce] allocating threads: 1
2015-02-12 16:12:48,724 INFO flow.FlowStep (BaseFlowStep.java:logInfo(834)) - [coerce] starting step: (1/1) ...persplatform/coercefields
2015-02-12 16:12:48,725 INFO jvm.JvmMetrics (JvmMetrics.java:init(71)) - Cannot initialize JVM Metrics with processName=JobTracker, sessionId= - already initialized
2015-02-12 16:12:48,725 INFO jvm.JvmMetrics (JvmMetrics.java:init(71)) - Cannot initialize JVM Metrics with processName=JobTracker, sessionId= - already initialized
2015-02-12 16:12:48,743 INFO mapred.FileInputFormat (FileInputFormat.java:listStatus(247)) - Total input paths to process : 1
2015-02-12 16:12:48,758 INFO mapreduce.JobSubmitter (JobSubmitter.java:submitJobInternal(492)) - number of splits:1
2015-02-12 16:12:48,770 INFO mapreduce.JobSubmitter (JobSubmitter.java:printTokens(575)) - Submitting tokens for job: job_local994598886_0011
2015-02-12 16:12:48,780 WARN conf.Configuration (Configuration.java:loadProperty(2510)) - file:/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/tmp/cascading/staging/soberoi994598886/.staging/job_local994598886_0011/job.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.retry.interval; Ignoring.
2015-02-12 16:12:48,781 WARN conf.Configuration (Configuration.java:loadProperty(2510)) - file:/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/tmp/cascading/staging/soberoi994598886/.staging/job_local994598886_0011/job.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.attempts; Ignoring.
2015-02-12 16:12:48,812 WARN conf.Configuration (Configuration.java:loadProperty(2510)) - file:/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/test/tmp/mapred/local/localRunner/soberoi/job_local994598886_0011/job_local994598886_0011.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.retry.interval; Ignoring.
2015-02-12 16:12:48,813 WARN conf.Configuration (Configuration.java:loadProperty(2510)) - file:/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/test/tmp/mapred/local/localRunner/soberoi/job_local994598886_0011/job_local994598886_0011.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.attempts; Ignoring.
2015-02-12 16:12:48,813 INFO mapreduce.Job (Job.java:submit(1299)) - The url to track the job: http://localhost:8080/
2015-02-12 16:12:48,813 INFO mapred.LocalJobRunner (LocalJobRunner.java:createOutputCommitter(471)) - OutputCommitter set in config null
2015-02-12 16:12:48,814 INFO flow.FlowStep (BaseFlowStep.java:logInfo(834)) - [coerce] submitted hadoop job: job_local994598886_0011
2015-02-12 16:12:48,814 INFO mapred.LocalJobRunner (LocalJobRunner.java:createOutputCommitter(489)) - OutputCommitter is org.apache.hadoop.mapred.FileOutputCommitter
2015-02-12 16:12:48,814 INFO flow.FlowStep (BaseFlowStep.java:logInfo(834)) - [coerce] tracking url: http://localhost:8080/
2015-02-12 16:12:48,815 INFO mapred.LocalJobRunner (LocalJobRunner.java:runTasks(448)) - Waiting for map tasks
2015-02-12 16:12:48,815 INFO mapred.LocalJobRunner (LocalJobRunner.java:run(224)) - Starting task: attempt_local994598886_0011_m_000000_0
2015-02-12 16:12:48,816 INFO util.ProcfsBasedProcessTree (ProcfsBasedProcessTree.java:isAvailable(181)) - ProcfsBasedProcessTree currently is supported only on Linux.
2015-02-12 16:12:48,816 INFO mapred.Task (Task.java:initialize(587)) - Using ResourceCalculatorProcessTree : null
2015-02-12 16:12:48,817 INFO io.MultiInputSplit (MultiInputSplit.java:readFields(161)) - current split input path: file:/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/classes/test/data/lower.txt
2015-02-12 16:12:48,817 INFO mapred.MapTask (MapTask.java:updateJobWithSplit(482)) - Processing split: cascading.tap.hadoop.io.MultiInputSplit@61a3e88e
2015-02-12 16:12:48,818 INFO mapred.MapTask (MapTask.java:runOldMapper(436)) - numReduceTasks: 0
2015-02-12 16:12:48,823 INFO hadoop.FlowMapper (FlowMapper.java:configure(71)) - cascading version: 2.6.3
2015-02-12 16:12:48,823 INFO hadoop.FlowMapper (FlowMapper.java:configure(72)) - child jvm opts: -Xmx200m
2015-02-12 16:12:48,827 INFO hadoop.FlowMapper (FlowMapper.java:configure(87)) - sourcing from: Hfs["TextLine[['offset', 'line']->[ALL]]"]["/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/classes/test/data/lower.txt"]
2015-02-12 16:12:48,828 INFO hadoop.FlowMapper (FlowMapper.java:configure(90)) - sinking to: Hfs["TextLine[['line']->['num', 'char']]"]["/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/test/output/hadoop2-mr1/assemblyhelpersplatform/coercefields"]
2015-02-12 16:12:48,833 INFO mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) -
2015-02-12 16:12:48,834 INFO mapred.Task (Task.java:done(1001)) - Task:attempt_local994598886_0011_m_000000_0 is done. And is in the process of committing
2015-02-12 16:12:48,835 INFO mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) -
2015-02-12 16:12:48,835 INFO mapred.Task (Task.java:commit(1162)) - Task attempt_local994598886_0011_m_000000_0 is allowed to commit now
2015-02-12 16:12:48,835 INFO output.FileOutputCommitter (FileOutputCommitter.java:commitTask(439)) - Saved output of task 'attempt_local994598886_0011_m_000000_0' to file:/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/test/output/hadoop2-mr1/assemblyhelpersplatform/coercefields/_temporary/0/task_local994598886_0011_m_000000
2015-02-12 16:12:48,836 INFO mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) - file:/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/classes/test/data/lower.txt:0+19
2015-02-12 16:12:48,836 INFO mapred.Task (Task.java:sendDone(1121)) - Task 'attempt_local994598886_0011_m_000000_0' done.
2015-02-12 16:12:48,836 INFO mapred.LocalJobRunner (LocalJobRunner.java:run(249)) - Finishing task: attempt_local994598886_0011_m_000000_0
2015-02-12 16:12:48,836 INFO mapred.LocalJobRunner (LocalJobRunner.java:runTasks(456)) - map task executor complete.
2015-02-12 16:12:48,854 INFO util.Hadoop18TapUtil (Hadoop18TapUtil.java:cleanTempPath(219)) - deleting temp path /Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/test/output/hadoop2-mr1/assemblyhelpersplatform/coercefields/_temporary
2015-02-12 16:12:48,873 INFO mapred.FileInputFormat (FileInputFormat.java:listStatus(247)) - Total input paths to process : 1
2015-02-12 16:12:48,874 INFO cascading.PlatformTestCase (PlatformTestCase.java:tearDown(209)) - copying to local /Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/test/output//hadoop2-mr1/assemblyhelpersplatform/coercefields
2015-02-12 16:12:48,887 INFO util.HadoopUtil (HadoopUtil.java:findMainClass(336)) - using default application jar, may cause class not found exceptions on the cluster
2015-02-12 16:12:48,887 INFO planner.HadoopPlanner (HadoopPlanner.java:initialize(225)) - using application jar: /Users/soberoi/.gradle/caches/modules-2/files-2.1/cascading/cascading-hadoop2-mr1/2.6.3/9ae016e2b8d246b9c1de9168e715d9f1b971aa4a/cascading-hadoop2-mr1-2.6.3.jar
2015-02-12 16:12:48,892 INFO Configuration.deprecation (Configuration.java:warnOnceIfDeprecated(1022)) - mapred.used.genericoptionsparser is deprecated. Instead, use mapreduce.client.genericoptionsparser.used
2015-02-12 16:12:48,894 INFO Configuration.deprecation (Configuration.java:warnOnceIfDeprecated(1022)) - mapred.job.tracker is deprecated. Instead, use mapreduce.jobtracker.address
2015-02-12 16:12:48,895 INFO Configuration.deprecation (Configuration.java:warnOnceIfDeprecated(1022)) - mapred.output.compress is deprecated. Instead, use mapreduce.output.fileoutputformat.compress
2015-02-12 16:12:48,900 INFO flow.Flow (BaseFlow.java:logInfo(1378)) - [count] starting
2015-02-12 16:12:48,900 INFO flow.Flow (BaseFlow.java:logInfo(1378)) - [count] source: Hfs["TextDelimited[['num', 'char']]"]["/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/classes/test/data/lhs.txt"]
2015-02-12 16:12:48,901 INFO flow.Flow (BaseFlow.java:logInfo(1378)) - [count] sink: Hfs["TextDelimited[['char', 'count']]"]["/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/test/output/hadoop2-mr1/assemblyhelpersplatform/count"]
2015-02-12 16:12:48,901 INFO flow.Flow (BaseFlow.java:logInfo(1378)) - [count] parallel execution is enabled: false
2015-02-12 16:12:48,901 INFO flow.Flow (BaseFlow.java:logInfo(1378)) - [count] starting jobs: 1
2015-02-12 16:12:48,901 INFO flow.Flow (BaseFlow.java:logInfo(1378)) - [count] allocating threads: 1
2015-02-12 16:12:48,901 INFO flow.FlowStep (BaseFlowStep.java:logInfo(834)) - [count] starting step: (1/1) ...mblyhelpersplatform/count
2015-02-12 16:12:48,902 INFO jvm.JvmMetrics (JvmMetrics.java:init(71)) - Cannot initialize JVM Metrics with processName=JobTracker, sessionId= - already initialized
2015-02-12 16:12:48,902 INFO jvm.JvmMetrics (JvmMetrics.java:init(71)) - Cannot initialize JVM Metrics with processName=JobTracker, sessionId= - already initialized
2015-02-12 16:12:48,921 INFO mapred.FileInputFormat (FileInputFormat.java:listStatus(247)) - Total input paths to process : 1
2015-02-12 16:12:48,937 INFO mapreduce.JobSubmitter (JobSubmitter.java:submitJobInternal(492)) - number of splits:1
2015-02-12 16:12:48,948 INFO mapreduce.JobSubmitter (JobSubmitter.java:printTokens(575)) - Submitting tokens for job: job_local910453495_0012
2015-02-12 16:12:48,958 WARN conf.Configuration (Configuration.java:loadProperty(2510)) - file:/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/tmp/cascading/staging/soberoi910453495/.staging/job_local910453495_0012/job.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.retry.interval; Ignoring.
2015-02-12 16:12:48,959 WARN conf.Configuration (Configuration.java:loadProperty(2510)) - file:/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/tmp/cascading/staging/soberoi910453495/.staging/job_local910453495_0012/job.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.attempts; Ignoring.
2015-02-12 16:12:48,989 WARN conf.Configuration (Configuration.java:loadProperty(2510)) - file:/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/test/tmp/mapred/local/localRunner/soberoi/job_local910453495_0012/job_local910453495_0012.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.retry.interval; Ignoring.
2015-02-12 16:12:48,990 WARN conf.Configuration (Configuration.java:loadProperty(2510)) - file:/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/test/tmp/mapred/local/localRunner/soberoi/job_local910453495_0012/job_local910453495_0012.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.attempts; Ignoring.
2015-02-12 16:12:48,990 INFO mapreduce.Job (Job.java:submit(1299)) - The url to track the job: http://localhost:8080/
2015-02-12 16:12:48,991 INFO mapred.LocalJobRunner (LocalJobRunner.java:createOutputCommitter(471)) - OutputCommitter set in config null
2015-02-12 16:12:48,991 INFO mapred.LocalJobRunner (LocalJobRunner.java:createOutputCommitter(489)) - OutputCommitter is org.apache.hadoop.mapred.FileOutputCommitter
2015-02-12 16:12:48,991 INFO flow.FlowStep (BaseFlowStep.java:logInfo(834)) - [count] submitted hadoop job: job_local910453495_0012
2015-02-12 16:12:48,991 INFO flow.FlowStep (BaseFlowStep.java:logInfo(834)) - [count] tracking url: http://localhost:8080/
2015-02-12 16:12:48,993 INFO mapred.LocalJobRunner (LocalJobRunner.java:runTasks(448)) - Waiting for map tasks
2015-02-12 16:12:48,993 INFO mapred.LocalJobRunner (LocalJobRunner.java:run(224)) - Starting task: attempt_local910453495_0012_m_000000_0
2015-02-12 16:12:48,994 INFO util.ProcfsBasedProcessTree (ProcfsBasedProcessTree.java:isAvailable(181)) - ProcfsBasedProcessTree currently is supported only on Linux.
2015-02-12 16:12:48,994 INFO mapred.Task (Task.java:initialize(587)) - Using ResourceCalculatorProcessTree : null
2015-02-12 16:12:48,994 INFO io.MultiInputSplit (MultiInputSplit.java:readFields(161)) - current split input path: file:/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/classes/test/data/lhs.txt
2015-02-12 16:12:48,994 INFO mapred.MapTask (MapTask.java:updateJobWithSplit(482)) - Processing split: cascading.tap.hadoop.io.MultiInputSplit@4db1f211
2015-02-12 16:12:48,995 INFO mapred.MapTask (MapTask.java:runOldMapper(436)) - numReduceTasks: 1
2015-02-12 16:12:49,012 INFO mapred.MapTask (MapTask.java:setEquator(1202)) - (EQUATOR) 0 kvi 26214396(104857584)
2015-02-12 16:12:49,012 INFO mapred.MapTask (MapTask.java:init(995)) - mapreduce.task.io.sort.mb: 100
2015-02-12 16:12:49,012 INFO mapred.MapTask (MapTask.java:init(996)) - soft limit at 83886080
2015-02-12 16:12:49,012 INFO mapred.MapTask (MapTask.java:init(997)) - bufstart = 0; bufvoid = 104857600
2015-02-12 16:12:49,013 INFO mapred.MapTask (MapTask.java:init(998)) - kvstart = 26214396; length = 6553600
2015-02-12 16:12:49,014 INFO mapred.MapTask (MapTask.java:createSortingCollector(402)) - Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer
2015-02-12 16:12:49,014 INFO hadoop.FlowMapper (FlowMapper.java:configure(71)) - cascading version: 2.6.3
2015-02-12 16:12:49,014 INFO hadoop.FlowMapper (FlowMapper.java:configure(72)) - child jvm opts: -Xmx200m
2015-02-12 16:12:49,019 INFO hadoop.FlowMapper (FlowMapper.java:configure(87)) - sourcing from: Hfs["TextDelimited[['num', 'char']]"]["/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/classes/test/data/lhs.txt"]
2015-02-12 16:12:49,019 INFO hadoop.FlowMapper (FlowMapper.java:configure(90)) - sinking to: GroupBy(count)[by:[{1}:'char']]
2015-02-12 16:12:49,019 INFO assembly.AggregateBy (AggregateBy.java:prepare(267)) - using threshold value: 2
2015-02-12 16:12:49,020 INFO assembly.AggregateBy (AggregateBy.java:removeEldestEntry(327)) - flushed keys num times: 1, with threshold: 2
2015-02-12 16:12:49,020 INFO assembly.AggregateBy (AggregateBy.java:removeEldestEntry(328)) - mem on flush (mb), free: 199, total: 509, max: 672
2015-02-12 16:12:49,020 INFO assembly.AggregateBy (AggregateBy.java:removeEldestEntry(333)) - total mem is 75% of max mem, to better utilize unused memory consider increasing current LRU threshold with system property "cascading.aggregateby.threshold"
2015-02-12 16:12:49,021 INFO assembly.AggregateBy (AggregateBy.java:removeEldestEntry(327)) - flushed keys num times: 3, with threshold: 2
2015-02-12 16:12:49,021 INFO assembly.AggregateBy (AggregateBy.java:removeEldestEntry(328)) - mem on flush (mb), free: 199, total: 509, max: 672
2015-02-12 16:12:49,021 INFO assembly.AggregateBy (AggregateBy.java:removeEldestEntry(333)) - total mem is 75% of max mem, to better utilize unused memory consider increasing current LRU threshold with system property "cascading.aggregateby.threshold"
2015-02-12 16:12:49,021 INFO assembly.AggregateBy (AggregateBy.java:removeEldestEntry(327)) - flushed keys num times: 5, with threshold: 2
2015-02-12 16:12:49,021 INFO assembly.AggregateBy (AggregateBy.java:removeEldestEntry(328)) - mem on flush (mb), free: 199, total: 509, max: 672
2015-02-12 16:12:49,021 INFO assembly.AggregateBy (AggregateBy.java:removeEldestEntry(333)) - total mem is 75% of max mem, to better utilize unused memory consider increasing current LRU threshold with system property "cascading.aggregateby.threshold"
2015-02-12 16:12:49,021 INFO assembly.AggregateBy (AggregateBy.java:removeEldestEntry(327)) - flushed keys num times: 7, with threshold: 2
2015-02-12 16:12:49,022 INFO assembly.AggregateBy (AggregateBy.java:removeEldestEntry(328)) - mem on flush (mb), free: 199, total: 509, max: 672
2015-02-12 16:12:49,022 INFO assembly.AggregateBy (AggregateBy.java:removeEldestEntry(333)) - total mem is 75% of max mem, to better utilize unused memory consider increasing current LRU threshold with system property "cascading.aggregateby.threshold"
2015-02-12 16:12:49,022 INFO mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) -
2015-02-12 16:12:49,023 INFO mapred.MapTask (MapTask.java:flush(1457)) - Starting flush of map output
2015-02-12 16:12:49,023 INFO mapred.MapTask (MapTask.java:flush(1475)) - Spilling map output
2015-02-12 16:12:49,023 INFO mapred.MapTask (MapTask.java:flush(1476)) - bufstart = 0; bufend = 99; bufvoid = 104857600
2015-02-12 16:12:49,023 INFO mapred.MapTask (MapTask.java:flush(1478)) - kvstart = 26214396(104857584); kvend = 26214364(104857456); length = 33/6553600
2015-02-12 16:12:49,024 INFO mapred.MapTask (MapTask.java:sortAndSpill(1660)) - Finished spill 0
2015-02-12 16:12:49,025 INFO mapred.Task (Task.java:done(1001)) - Task:attempt_local910453495_0012_m_000000_0 is done. And is in the process of committing
2015-02-12 16:12:49,027 INFO mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) - file:/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/classes/test/data/lhs.txt:0+52
2015-02-12 16:12:49,027 INFO mapred.Task (Task.java:sendDone(1121)) - Task 'attempt_local910453495_0012_m_000000_0' done.
2015-02-12 16:12:49,027 INFO mapred.LocalJobRunner (LocalJobRunner.java:run(249)) - Finishing task: attempt_local910453495_0012_m_000000_0
2015-02-12 16:12:49,027 INFO mapred.LocalJobRunner (LocalJobRunner.java:runTasks(456)) - map task executor complete.
2015-02-12 16:12:49,027 INFO mapred.LocalJobRunner (LocalJobRunner.java:runTasks(448)) - Waiting for reduce tasks
2015-02-12 16:12:49,027 INFO mapred.LocalJobRunner (LocalJobRunner.java:run(302)) - Starting task: attempt_local910453495_0012_r_000000_0
2015-02-12 16:12:49,028 INFO util.ProcfsBasedProcessTree (ProcfsBasedProcessTree.java:isAvailable(181)) - ProcfsBasedProcessTree currently is supported only on Linux.
2015-02-12 16:12:49,029 INFO mapred.Task (Task.java:initialize(587)) - Using ResourceCalculatorProcessTree : null
2015-02-12 16:12:49,029 INFO mapred.ReduceTask (ReduceTask.java:run(362)) - Using ShuffleConsumerPlugin: org.apache.hadoop.mapreduce.task.reduce.Shuffle@202a6eeb
2015-02-12 16:12:49,029 INFO reduce.MergeManagerImpl (MergeManagerImpl.java:<init>(195)) - MergerManager: memoryLimit=493250144, maxSingleShuffleLimit=123312536, mergeThreshold=325545120, ioSortFactor=10, memToMemMergeOutputsThreshold=10
2015-02-12 16:12:49,030 INFO reduce.EventFetcher (EventFetcher.java:run(61)) - attempt_local910453495_0012_r_000000_0 Thread started: EventFetcher for fetching Map Completion Events
2015-02-12 16:12:49,030 INFO reduce.LocalFetcher (LocalFetcher.java:copyMapOutput(141)) - localfetcher#10 about to shuffle output of map attempt_local910453495_0012_m_000000_0 decomp: 119 len: 123 to MEMORY
2015-02-12 16:12:49,031 INFO reduce.InMemoryMapOutput (InMemoryMapOutput.java:shuffle(100)) - Read 119 bytes from map-output for attempt_local910453495_0012_m_000000_0
2015-02-12 16:12:49,031 INFO reduce.MergeManagerImpl (MergeManagerImpl.java:closeInMemoryFile(313)) - closeInMemoryFile -> map-output of size: 119, inMemoryMapOutputs.size() -> 1, commitMemory -> 0, usedMemory ->119
2015-02-12 16:12:49,031 INFO reduce.EventFetcher (EventFetcher.java:run(76)) - EventFetcher is interrupted.. Returning
2015-02-12 16:12:49,032 INFO mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) - 1 / 1 copied.
2015-02-12 16:12:49,032 INFO reduce.MergeManagerImpl (MergeManagerImpl.java:finalMerge(673)) - finalMerge called with 1 in-memory map-outputs and 0 on-disk map-outputs
2015-02-12 16:12:49,033 INFO mapred.Merger (Merger.java:merge(606)) - Merging 1 sorted segments
2015-02-12 16:12:49,033 INFO mapred.Merger (Merger.java:merge(705)) - Down to the last merge-pass, with 1 segments left of total size: 110 bytes
2015-02-12 16:12:49,034 INFO reduce.MergeManagerImpl (MergeManagerImpl.java:finalMerge(750)) - Merged 1 segments, 119 bytes to disk to satisfy reduce memory limit
2015-02-12 16:12:49,034 INFO reduce.MergeManagerImpl (MergeManagerImpl.java:finalMerge(780)) - Merging 1 files, 123 bytes from disk
2015-02-12 16:12:49,034 INFO reduce.MergeManagerImpl (MergeManagerImpl.java:finalMerge(795)) - Merging 0 segments, 0 bytes from memory into reduce
2015-02-12 16:12:49,034 INFO mapred.Merger (Merger.java:merge(606)) - Merging 1 sorted segments
2015-02-12 16:12:49,035 INFO mapred.Merger (Merger.java:merge(705)) - Down to the last merge-pass, with 1 segments left of total size: 110 bytes
2015-02-12 16:12:49,035 INFO mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) - 1 / 1 copied.
2015-02-12 16:12:49,035 INFO hadoop.FlowReducer (FlowReducer.java:configure(78)) - cascading version: 2.6.3
2015-02-12 16:12:49,036 INFO hadoop.FlowReducer (FlowReducer.java:configure(79)) - child jvm opts: -Xmx200m
2015-02-12 16:12:49,042 INFO hadoop.FlowReducer (FlowReducer.java:configure(97)) - sourcing from: GroupBy(count)[by:[{1}:'char']]
2015-02-12 16:12:49,042 INFO hadoop.FlowReducer (FlowReducer.java:configure(100)) - sinking to: Hfs["TextDelimited[['char', 'count']]"]["/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/test/output/hadoop2-mr1/assemblyhelpersplatform/count"]
2015-02-12 16:12:49,049 INFO mapred.Task (Task.java:done(1001)) - Task:attempt_local910453495_0012_r_000000_0 is done. And is in the process of committing
2015-02-12 16:12:49,050 INFO mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) - 1 / 1 copied.
2015-02-12 16:12:49,050 INFO mapred.Task (Task.java:commit(1162)) - Task attempt_local910453495_0012_r_000000_0 is allowed to commit now
2015-02-12 16:12:49,051 INFO output.FileOutputCommitter (FileOutputCommitter.java:commitTask(439)) - Saved output of task 'attempt_local910453495_0012_r_000000_0' to file:/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/test/output/hadoop2-mr1/assemblyhelpersplatform/count/_temporary/0/task_local910453495_0012_r_000000
2015-02-12 16:12:49,052 INFO mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) - reduce > reduce
2015-02-12 16:12:49,052 INFO mapred.Task (Task.java:sendDone(1121)) - Task 'attempt_local910453495_0012_r_000000_0' done.
2015-02-12 16:12:49,052 INFO mapred.LocalJobRunner (LocalJobRunner.java:run(325)) - Finishing task: attempt_local910453495_0012_r_000000_0
2015-02-12 16:12:49,052 INFO mapred.LocalJobRunner (LocalJobRunner.java:runTasks(456)) - reduce task executor complete.
2015-02-12 16:12:49,070 INFO util.Hadoop18TapUtil (Hadoop18TapUtil.java:cleanTempPath(219)) - deleting temp path /Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/test/output/hadoop2-mr1/assemblyhelpersplatform/count/_temporary
2015-02-12 16:12:49,091 INFO mapred.FileInputFormat (FileInputFormat.java:listStatus(247)) - Total input paths to process : 1
2015-02-12 16:12:49,113 INFO mapred.FileInputFormat (FileInputFormat.java:listStatus(247)) - Total input paths to process : 1
2015-02-12 16:12:49,114 INFO cascading.PlatformTestCase (PlatformTestCase.java:tearDown(209)) - copying to local /Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/test/output//hadoop2-mr1/assemblyhelpersplatform/count
2015-02-12 16:12:49,132 INFO util.HadoopUtil (HadoopUtil.java:findMainClass(336)) - using default application jar, may cause class not found exceptions on the cluster
2015-02-12 16:12:49,132 INFO planner.HadoopPlanner (HadoopPlanner.java:initialize(225)) - using application jar: /Users/soberoi/.gradle/caches/modules-2/files-2.1/cascading/cascading-hadoop2-mr1/2.6.3/9ae016e2b8d246b9c1de9168e715d9f1b971aa4a/cascading-hadoop2-mr1-2.6.3.jar
2015-02-12 16:12:49,137 INFO Configuration.deprecation (Configuration.java:warnOnceIfDeprecated(1022)) - mapred.used.genericoptionsparser is deprecated. Instead, use mapreduce.client.genericoptionsparser.used
2015-02-12 16:12:49,138 INFO Configuration.deprecation (Configuration.java:warnOnceIfDeprecated(1022)) - mapred.job.tracker is deprecated. Instead, use mapreduce.jobtracker.address
2015-02-12 16:12:49,138 INFO Configuration.deprecation (Configuration.java:warnOnceIfDeprecated(1022)) - mapred.output.compress is deprecated. Instead, use mapreduce.output.fileoutputformat.compress
2015-02-12 16:12:49,143 INFO flow.Flow (BaseFlow.java:logInfo(1378)) - [max] starting
2015-02-12 16:12:49,143 INFO flow.Flow (BaseFlow.java:logInfo(1378)) - [max] source: Hfs["TextDelimited[['num', 'char']]"]["/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/classes/test/data/lhs.txt"]
2015-02-12 16:12:49,143 INFO flow.Flow (BaseFlow.java:logInfo(1378)) - [max] sink: Hfs["TextDelimited[['char', 'max']]"]["/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/test/output/hadoop2-mr1/assemblyhelpersplatform/maxby"]
2015-02-12 16:12:49,143 INFO flow.Flow (BaseFlow.java:logInfo(1378)) - [max] parallel execution is enabled: false
2015-02-12 16:12:49,144 INFO flow.Flow (BaseFlow.java:logInfo(1378)) - [max] starting jobs: 1
2015-02-12 16:12:49,144 INFO flow.Flow (BaseFlow.java:logInfo(1378)) - [max] allocating threads: 1
2015-02-12 16:12:49,144 INFO flow.FlowStep (BaseFlowStep.java:logInfo(834)) - [max] starting step: (1/1) ...mblyhelpersplatform/maxby
2015-02-12 16:12:49,145 INFO jvm.JvmMetrics (JvmMetrics.java:init(71)) - Cannot initialize JVM Metrics with processName=JobTracker, sessionId= - already initialized
2015-02-12 16:12:49,146 INFO jvm.JvmMetrics (JvmMetrics.java:init(71)) - Cannot initialize JVM Metrics with processName=JobTracker, sessionId= - already initialized
2015-02-12 16:12:49,167 INFO mapred.FileInputFormat (FileInputFormat.java:listStatus(247)) - Total input paths to process : 1
2015-02-12 16:12:49,185 INFO mapreduce.JobSubmitter (JobSubmitter.java:submitJobInternal(492)) - number of splits:1
2015-02-12 16:12:49,197 INFO mapreduce.JobSubmitter (JobSubmitter.java:printTokens(575)) - Submitting tokens for job: job_local1579825300_0013
2015-02-12 16:12:49,213 WARN conf.Configuration (Configuration.java:loadProperty(2510)) - file:/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/tmp/cascading/staging/soberoi1579825300/.staging/job_local1579825300_0013/job.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.retry.interval; Ignoring.
2015-02-12 16:12:49,214 WARN conf.Configuration (Configuration.java:loadProperty(2510)) - file:/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/tmp/cascading/staging/soberoi1579825300/.staging/job_local1579825300_0013/job.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.attempts; Ignoring.
2015-02-12 16:12:49,247 WARN conf.Configuration (Configuration.java:loadProperty(2510)) - file:/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/test/tmp/mapred/local/localRunner/soberoi/job_local1579825300_0013/job_local1579825300_0013.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.retry.interval; Ignoring.
2015-02-12 16:12:49,248 WARN conf.Configuration (Configuration.java:loadProperty(2510)) - file:/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/test/tmp/mapred/local/localRunner/soberoi/job_local1579825300_0013/job_local1579825300_0013.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.attempts; Ignoring.
2015-02-12 16:12:49,248 INFO mapreduce.Job (Job.java:submit(1299)) - The url to track the job: http://localhost:8080/
2015-02-12 16:12:49,249 INFO mapred.LocalJobRunner (LocalJobRunner.java:createOutputCommitter(471)) - OutputCommitter set in config null
2015-02-12 16:12:49,249 INFO flow.FlowStep (BaseFlowStep.java:logInfo(834)) - [max] submitted hadoop job: job_local1579825300_0013
2015-02-12 16:12:49,249 INFO mapred.LocalJobRunner (LocalJobRunner.java:createOutputCommitter(489)) - OutputCommitter is org.apache.hadoop.mapred.FileOutputCommitter
2015-02-12 16:12:49,249 INFO flow.FlowStep (BaseFlowStep.java:logInfo(834)) - [max] tracking url: http://localhost:8080/
2015-02-12 16:12:49,250 INFO mapred.LocalJobRunner (LocalJobRunner.java:runTasks(448)) - Waiting for map tasks
2015-02-12 16:12:49,251 INFO mapred.LocalJobRunner (LocalJobRunner.java:run(224)) - Starting task: attempt_local1579825300_0013_m_000000_0
2015-02-12 16:12:49,251 INFO util.ProcfsBasedProcessTree (ProcfsBasedProcessTree.java:isAvailable(181)) - ProcfsBasedProcessTree currently is supported only on Linux.
2015-02-12 16:12:49,252 INFO mapred.Task (Task.java:initialize(587)) - Using ResourceCalculatorProcessTree : null
2015-02-12 16:12:49,253 INFO io.MultiInputSplit (MultiInputSplit.java:readFields(161)) - current split input path: file:/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/classes/test/data/lhs.txt
2015-02-12 16:12:49,253 INFO mapred.MapTask (MapTask.java:updateJobWithSplit(482)) - Processing split: cascading.tap.hadoop.io.MultiInputSplit@67825189
2015-02-12 16:12:49,254 INFO mapred.MapTask (MapTask.java:runOldMapper(436)) - numReduceTasks: 1
2015-02-12 16:12:49,270 INFO mapred.MapTask (MapTask.java:setEquator(1202)) - (EQUATOR) 0 kvi 26214396(104857584)
2015-02-12 16:12:49,271 INFO mapred.MapTask (MapTask.java:init(995)) - mapreduce.task.io.sort.mb: 100
2015-02-12 16:12:49,271 INFO mapred.MapTask (MapTask.java:init(996)) - soft limit at 83886080
2015-02-12 16:12:49,271 INFO mapred.MapTask (MapTask.java:init(997)) - bufstart = 0; bufvoid = 104857600
2015-02-12 16:12:49,271 INFO mapred.MapTask (MapTask.java:init(998)) - kvstart = 26214396; length = 6553600
2015-02-12 16:12:49,272 INFO mapred.MapTask (MapTask.java:createSortingCollector(402)) - Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer
2015-02-12 16:12:49,272 INFO hadoop.FlowMapper (FlowMapper.java:configure(71)) - cascading version: 2.6.3
2015-02-12 16:12:49,272 INFO hadoop.FlowMapper (FlowMapper.java:configure(72)) - child jvm opts: -Xmx200m
2015-02-12 16:12:49,277 INFO hadoop.FlowMapper (FlowMapper.java:configure(87)) - sourcing from: Hfs["TextDelimited[['num', 'char']]"]["/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/classes/test/data/lhs.txt"]
2015-02-12 16:12:49,277 INFO hadoop.FlowMapper (FlowMapper.java:configure(90)) - sinking to: GroupBy(max)[by:[{1}:'char']]
2015-02-12 16:12:49,277 INFO assembly.AggregateBy (AggregateBy.java:prepare(267)) - using threshold value: 2
2015-02-12 16:12:49,278 INFO assembly.AggregateBy (AggregateBy.java:removeEldestEntry(327)) - flushed keys num times: 1, with threshold: 2
2015-02-12 16:12:49,278 INFO assembly.AggregateBy (AggregateBy.java:removeEldestEntry(328)) - mem on flush (mb), free: 199, total: 509, max: 672
2015-02-12 16:12:49,278 INFO assembly.AggregateBy (AggregateBy.java:removeEldestEntry(333)) - total mem is 75% of max mem, to better utilize unused memory consider increasing current LRU threshold with system property "cascading.aggregateby.threshold"
2015-02-12 16:12:49,279 INFO assembly.AggregateBy (AggregateBy.java:removeEldestEntry(327)) - flushed keys num times: 3, with threshold: 2
2015-02-12 16:12:49,279 INFO assembly.AggregateBy (AggregateBy.java:removeEldestEntry(328)) - mem on flush (mb), free: 199, total: 509, max: 672
2015-02-12 16:12:49,279 INFO assembly.AggregateBy (AggregateBy.java:removeEldestEntry(333)) - total mem is 75% of max mem, to better utilize unused memory consider increasing current LRU threshold with system property "cascading.aggregateby.threshold"
2015-02-12 16:12:49,280 INFO assembly.AggregateBy (AggregateBy.java:removeEldestEntry(327)) - flushed keys num times: 5, with threshold: 2
2015-02-12 16:12:49,280 INFO assembly.AggregateBy (AggregateBy.java:removeEldestEntry(328)) - mem on flush (mb), free: 199, total: 509, max: 672
2015-02-12 16:12:49,280 INFO assembly.AggregateBy (AggregateBy.java:removeEldestEntry(333)) - total mem is 75% of max mem, to better utilize unused memory consider increasing current LRU threshold with system property "cascading.aggregateby.threshold"
2015-02-12 16:12:49,280 INFO assembly.AggregateBy (AggregateBy.java:removeEldestEntry(327)) - flushed keys num times: 7, with threshold: 2
2015-02-12 16:12:49,280 INFO assembly.AggregateBy (AggregateBy.java:removeEldestEntry(328)) - mem on flush (mb), free: 199, total: 509, max: 672
2015-02-12 16:12:49,280 INFO assembly.AggregateBy (AggregateBy.java:removeEldestEntry(333)) - total mem is 75% of max mem, to better utilize unused memory consider increasing current LRU threshold with system property "cascading.aggregateby.threshold"
2015-02-12 16:12:49,281 INFO mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) -
2015-02-12 16:12:49,281 INFO mapred.MapTask (MapTask.java:flush(1457)) - Starting flush of map output
2015-02-12 16:12:49,281 INFO mapred.MapTask (MapTask.java:flush(1475)) - Spilling map output
2015-02-12 16:12:49,282 INFO mapred.MapTask (MapTask.java:flush(1476)) - bufstart = 0; bufend = 135; bufvoid = 104857600
2015-02-12 16:12:49,282 INFO mapred.MapTask (MapTask.java:flush(1478)) - kvstart = 26214396(104857584); kvend = 26214364(104857456); length = 33/6553600
2015-02-12 16:12:49,283 INFO mapred.MapTask (MapTask.java:sortAndSpill(1660)) - Finished spill 0
2015-02-12 16:12:49,285 INFO mapred.Task (Task.java:done(1001)) - Task:attempt_local1579825300_0013_m_000000_0 is done. And is in the process of committing
2015-02-12 16:12:49,286 INFO mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) - file:/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/classes/test/data/lhs.txt:0+52
2015-02-12 16:12:49,286 INFO mapred.Task (Task.java:sendDone(1121)) - Task 'attempt_local1579825300_0013_m_000000_0' done.
2015-02-12 16:12:49,286 INFO mapred.LocalJobRunner (LocalJobRunner.java:run(249)) - Finishing task: attempt_local1579825300_0013_m_000000_0
2015-02-12 16:12:49,286 INFO mapred.LocalJobRunner (LocalJobRunner.java:runTasks(456)) - map task executor complete.
2015-02-12 16:12:49,287 INFO mapred.LocalJobRunner (LocalJobRunner.java:runTasks(448)) - Waiting for reduce tasks
2015-02-12 16:12:49,287 INFO mapred.LocalJobRunner (LocalJobRunner.java:run(302)) - Starting task: attempt_local1579825300_0013_r_000000_0
2015-02-12 16:12:49,288 INFO util.ProcfsBasedProcessTree (ProcfsBasedProcessTree.java:isAvailable(181)) - ProcfsBasedProcessTree currently is supported only on Linux.
2015-02-12 16:12:49,288 INFO mapred.Task (Task.java:initialize(587)) - Using ResourceCalculatorProcessTree : null
2015-02-12 16:12:49,288 INFO mapred.ReduceTask (ReduceTask.java:run(362)) - Using ShuffleConsumerPlugin: org.apache.hadoop.mapreduce.task.reduce.Shuffle@13233788
2015-02-12 16:12:49,289 INFO reduce.MergeManagerImpl (MergeManagerImpl.java:<init>(195)) - MergerManager: memoryLimit=493250144, maxSingleShuffleLimit=123312536, mergeThreshold=325545120, ioSortFactor=10, memToMemMergeOutputsThreshold=10
2015-02-12 16:12:49,289 INFO reduce.EventFetcher (EventFetcher.java:run(61)) - attempt_local1579825300_0013_r_000000_0 Thread started: EventFetcher for fetching Map Completion Events
2015-02-12 16:12:49,290 INFO reduce.LocalFetcher (LocalFetcher.java:copyMapOutput(141)) - localfetcher#11 about to shuffle output of map attempt_local1579825300_0013_m_000000_0 decomp: 155 len: 159 to MEMORY
2015-02-12 16:12:49,291 INFO reduce.InMemoryMapOutput (InMemoryMapOutput.java:shuffle(100)) - Read 155 bytes from map-output for attempt_local1579825300_0013_m_000000_0
2015-02-12 16:12:49,291 INFO reduce.MergeManagerImpl (MergeManagerImpl.java:closeInMemoryFile(313)) - closeInMemoryFile -> map-output of size: 155, inMemoryMapOutputs.size() -> 1, commitMemory -> 0, usedMemory ->155
2015-02-12 16:12:49,291 INFO reduce.EventFetcher (EventFetcher.java:run(76)) - EventFetcher is interrupted.. Returning
2015-02-12 16:12:49,292 INFO mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) - 1 / 1 copied.
2015-02-12 16:12:49,292 INFO reduce.MergeManagerImpl (MergeManagerImpl.java:finalMerge(673)) - finalMerge called with 1 in-memory map-outputs and 0 on-disk map-outputs
2015-02-12 16:12:49,293 INFO mapred.Merger (Merger.java:merge(606)) - Merging 1 sorted segments
2015-02-12 16:12:49,293 INFO mapred.Merger (Merger.java:merge(705)) - Down to the last merge-pass, with 1 segments left of total size: 146 bytes
2015-02-12 16:12:49,294 INFO reduce.MergeManagerImpl (MergeManagerImpl.java:finalMerge(750)) - Merged 1 segments, 155 bytes to disk to satisfy reduce memory limit
2015-02-12 16:12:49,294 INFO reduce.MergeManagerImpl (MergeManagerImpl.java:finalMerge(780)) - Merging 1 files, 159 bytes from disk
2015-02-12 16:12:49,294 INFO reduce.MergeManagerImpl (MergeManagerImpl.java:finalMerge(795)) - Merging 0 segments, 0 bytes from memory into reduce
2015-02-12 16:12:49,295 INFO mapred.Merger (Merger.java:merge(606)) - Merging 1 sorted segments
2015-02-12 16:12:49,295 INFO mapred.Merger (Merger.java:merge(705)) - Down to the last merge-pass, with 1 segments left of total size: 146 bytes
2015-02-12 16:12:49,295 INFO mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) - 1 / 1 copied.
2015-02-12 16:12:49,296 INFO hadoop.FlowReducer (FlowReducer.java:configure(78)) - cascading version: 2.6.3
2015-02-12 16:12:49,296 INFO hadoop.FlowReducer (FlowReducer.java:configure(79)) - child jvm opts: -Xmx200m
2015-02-12 16:12:49,301 INFO hadoop.FlowReducer (FlowReducer.java:configure(97)) - sourcing from: GroupBy(max)[by:[{1}:'char']]
2015-02-12 16:12:49,301 INFO hadoop.FlowReducer (FlowReducer.java:configure(100)) - sinking to: Hfs["TextDelimited[['char', 'max']]"]["/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/test/output/hadoop2-mr1/assemblyhelpersplatform/maxby"]
2015-02-12 16:12:49,309 INFO mapred.Task (Task.java:done(1001)) - Task:attempt_local1579825300_0013_r_000000_0 is done. And is in the process of committing
2015-02-12 16:12:49,310 INFO mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) - 1 / 1 copied.
2015-02-12 16:12:49,310 INFO mapred.Task (Task.java:commit(1162)) - Task attempt_local1579825300_0013_r_000000_0 is allowed to commit now
2015-02-12 16:12:49,310 INFO output.FileOutputCommitter (FileOutputCommitter.java:commitTask(439)) - Saved output of task 'attempt_local1579825300_0013_r_000000_0' to file:/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/test/output/hadoop2-mr1/assemblyhelpersplatform/maxby/_temporary/0/task_local1579825300_0013_r_000000
2015-02-12 16:12:49,311 INFO mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) - reduce > reduce
2015-02-12 16:12:49,311 INFO mapred.Task (Task.java:sendDone(1121)) - Task 'attempt_local1579825300_0013_r_000000_0' done.
2015-02-12 16:12:49,311 INFO mapred.LocalJobRunner (LocalJobRunner.java:run(325)) - Finishing task: attempt_local1579825300_0013_r_000000_0
2015-02-12 16:12:49,312 INFO mapred.LocalJobRunner (LocalJobRunner.java:runTasks(456)) - reduce task executor complete.
2015-02-12 16:12:49,322 INFO util.Hadoop18TapUtil (Hadoop18TapUtil.java:cleanTempPath(219)) - deleting temp path /Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/test/output/hadoop2-mr1/assemblyhelpersplatform/maxby/_temporary
2015-02-12 16:12:49,343 INFO mapred.FileInputFormat (FileInputFormat.java:listStatus(247)) - Total input paths to process : 1
2015-02-12 16:12:49,364 INFO mapred.FileInputFormat (FileInputFormat.java:listStatus(247)) - Total input paths to process : 1
2015-02-12 16:12:49,365 INFO cascading.PlatformTestCase (PlatformTestCase.java:tearDown(209)) - copying to local /Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/test/output//hadoop2-mr1/assemblyhelpersplatform/maxby
2015-02-12 16:12:49,377 INFO util.HadoopUtil (HadoopUtil.java:findMainClass(336)) - using default application jar, may cause class not found exceptions on the cluster
2015-02-12 16:12:49,377 INFO planner.HadoopPlanner (HadoopPlanner.java:initialize(225)) - using application jar: /Users/soberoi/.gradle/caches/modules-2/files-2.1/cascading/cascading-hadoop2-mr1/2.6.3/9ae016e2b8d246b9c1de9168e715d9f1b971aa4a/cascading-hadoop2-mr1-2.6.3.jar
2015-02-12 16:12:49,382 INFO Configuration.deprecation (Configuration.java:warnOnceIfDeprecated(1022)) - mapred.used.genericoptionsparser is deprecated. Instead, use mapreduce.client.genericoptionsparser.used
2015-02-12 16:12:49,383 INFO Configuration.deprecation (Configuration.java:warnOnceIfDeprecated(1022)) - mapred.job.tracker is deprecated. Instead, use mapreduce.jobtracker.address
2015-02-12 16:12:49,383 INFO Configuration.deprecation (Configuration.java:warnOnceIfDeprecated(1022)) - mapred.output.compress is deprecated. Instead, use mapreduce.output.fileoutputformat.compress
2015-02-12 16:12:49,388 INFO flow.Flow (BaseFlow.java:logInfo(1378)) - [min] starting
2015-02-12 16:12:49,388 INFO flow.Flow (BaseFlow.java:logInfo(1378)) - [min] source: Hfs["TextDelimited[['num', 'char']]"]["/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/classes/test/data/lhs.txt"]
2015-02-12 16:12:49,388 INFO flow.Flow (BaseFlow.java:logInfo(1378)) - [min] sink: Hfs["TextDelimited[['char', 'min']]"]["/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/test/output/hadoop2-mr1/assemblyhelpersplatform/minby"]
2015-02-12 16:12:49,388 INFO flow.Flow (BaseFlow.java:logInfo(1378)) - [min] parallel execution is enabled: false
2015-02-12 16:12:49,388 INFO flow.Flow (BaseFlow.java:logInfo(1378)) - [min] starting jobs: 1
2015-02-12 16:12:49,388 INFO flow.Flow (BaseFlow.java:logInfo(1378)) - [min] allocating threads: 1
2015-02-12 16:12:49,388 INFO flow.FlowStep (BaseFlowStep.java:logInfo(834)) - [min] starting step: (1/1) ...mblyhelpersplatform/minby
2015-02-12 16:12:49,389 INFO jvm.JvmMetrics (JvmMetrics.java:init(71)) - Cannot initialize JVM Metrics with processName=JobTracker, sessionId= - already initialized
2015-02-12 16:12:49,389 INFO jvm.JvmMetrics (JvmMetrics.java:init(71)) - Cannot initialize JVM Metrics with processName=JobTracker, sessionId= - already initialized
2015-02-12 16:12:49,407 INFO mapred.FileInputFormat (FileInputFormat.java:listStatus(247)) - Total input paths to process : 1
2015-02-12 16:12:49,423 INFO mapreduce.JobSubmitter (JobSubmitter.java:submitJobInternal(492)) - number of splits:1
2015-02-12 16:12:49,434 INFO mapreduce.JobSubmitter (JobSubmitter.java:printTokens(575)) - Submitting tokens for job: job_local1907556576_0014
2015-02-12 16:12:49,444 WARN conf.Configuration (Configuration.java:loadProperty(2510)) - file:/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/tmp/cascading/staging/soberoi1907556576/.staging/job_local1907556576_0014/job.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.retry.interval; Ignoring.
2015-02-12 16:12:49,445 WARN conf.Configuration (Configuration.java:loadProperty(2510)) - file:/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/tmp/cascading/staging/soberoi1907556576/.staging/job_local1907556576_0014/job.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.attempts; Ignoring.
2015-02-12 16:12:49,476 WARN conf.Configuration (Configuration.java:loadProperty(2510)) - file:/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/test/tmp/mapred/local/localRunner/soberoi/job_local1907556576_0014/job_local1907556576_0014.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.retry.interval; Ignoring.
2015-02-12 16:12:49,477 WARN conf.Configuration (Configuration.java:loadProperty(2510)) - file:/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/test/tmp/mapred/local/localRunner/soberoi/job_local1907556576_0014/job_local1907556576_0014.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.attempts; Ignoring.
2015-02-12 16:12:49,477 INFO mapreduce.Job (Job.java:submit(1299)) - The url to track the job: http://localhost:8080/
2015-02-12 16:12:49,478 INFO mapred.LocalJobRunner (LocalJobRunner.java:createOutputCommitter(471)) - OutputCommitter set in config null
2015-02-12 16:12:49,478 INFO flow.FlowStep (BaseFlowStep.java:logInfo(834)) - [min] submitted hadoop job: job_local1907556576_0014
2015-02-12 16:12:49,478 INFO mapred.LocalJobRunner (LocalJobRunner.java:createOutputCommitter(489)) - OutputCommitter is org.apache.hadoop.mapred.FileOutputCommitter
2015-02-12 16:12:49,478 INFO flow.FlowStep (BaseFlowStep.java:logInfo(834)) - [min] tracking url: http://localhost:8080/
2015-02-12 16:12:49,480 INFO mapred.LocalJobRunner (LocalJobRunner.java:runTasks(448)) - Waiting for map tasks
2015-02-12 16:12:49,480 INFO mapred.LocalJobRunner (LocalJobRunner.java:run(224)) - Starting task: attempt_local1907556576_0014_m_000000_0
2015-02-12 16:12:49,481 INFO util.ProcfsBasedProcessTree (ProcfsBasedProcessTree.java:isAvailable(181)) - ProcfsBasedProcessTree currently is supported only on Linux.
2015-02-12 16:12:49,481 INFO mapred.Task (Task.java:initialize(587)) - Using ResourceCalculatorProcessTree : null
2015-02-12 16:12:49,482 INFO io.MultiInputSplit (MultiInputSplit.java:readFields(161)) - current split input path: file:/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/classes/test/data/lhs.txt
2015-02-12 16:12:49,483 INFO mapred.MapTask (MapTask.java:updateJobWithSplit(482)) - Processing split: cascading.tap.hadoop.io.MultiInputSplit@367019ee
2015-02-12 16:12:49,484 INFO mapred.MapTask (MapTask.java:runOldMapper(436)) - numReduceTasks: 1
2015-02-12 16:12:49,499 INFO mapred.MapTask (MapTask.java:setEquator(1202)) - (EQUATOR) 0 kvi 26214396(104857584)
2015-02-12 16:12:49,500 INFO mapred.MapTask (MapTask.java:init(995)) - mapreduce.task.io.sort.mb: 100
2015-02-12 16:12:49,500 INFO mapred.MapTask (MapTask.java:init(996)) - soft limit at 83886080
2015-02-12 16:12:49,500 INFO mapred.MapTask (MapTask.java:init(997)) - bufstart = 0; bufvoid = 104857600
2015-02-12 16:12:49,500 INFO mapred.MapTask (MapTask.java:init(998)) - kvstart = 26214396; length = 6553600
2015-02-12 16:12:49,502 INFO mapred.MapTask (MapTask.java:createSortingCollector(402)) - Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer
2015-02-12 16:12:49,502 INFO hadoop.FlowMapper (FlowMapper.java:configure(71)) - cascading version: 2.6.3
2015-02-12 16:12:49,502 INFO hadoop.FlowMapper (FlowMapper.java:configure(72)) - child jvm opts: -Xmx200m
2015-02-12 16:12:49,509 INFO hadoop.FlowMapper (FlowMapper.java:configure(87)) - sourcing from: Hfs["TextDelimited[['num', 'char']]"]["/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/classes/test/data/lhs.txt"]
2015-02-12 16:12:49,509 INFO hadoop.FlowMapper (FlowMapper.java:configure(90)) - sinking to: GroupBy(min)[by:[{1}:'char']]
2015-02-12 16:12:49,510 INFO assembly.AggregateBy (AggregateBy.java:prepare(267)) - using threshold value: 2
2015-02-12 16:12:49,511 INFO assembly.AggregateBy (AggregateBy.java:removeEldestEntry(327)) - flushed keys num times: 1, with threshold: 2
2015-02-12 16:12:49,511 INFO assembly.AggregateBy (AggregateBy.java:removeEldestEntry(328)) - mem on flush (mb), free: 187, total: 498, max: 672
2015-02-12 16:12:49,511 INFO assembly.AggregateBy (AggregateBy.java:removeEldestEntry(333)) - total mem is 74% of max mem, to better utilize unused memory consider increasing current LRU threshold with system property "cascading.aggregateby.threshold"
2015-02-12 16:12:49,512 INFO assembly.AggregateBy (AggregateBy.java:removeEldestEntry(327)) - flushed keys num times: 3, with threshold: 2
2015-02-12 16:12:49,512 INFO assembly.AggregateBy (AggregateBy.java:removeEldestEntry(328)) - mem on flush (mb), free: 187, total: 498, max: 672
2015-02-12 16:12:49,512 INFO assembly.AggregateBy (AggregateBy.java:removeEldestEntry(333)) - total mem is 74% of max mem, to better utilize unused memory consider increasing current LRU threshold with system property "cascading.aggregateby.threshold"
2015-02-12 16:12:49,512 INFO assembly.AggregateBy (AggregateBy.java:removeEldestEntry(327)) - flushed keys num times: 5, with threshold: 2
2015-02-12 16:12:49,513 INFO assembly.AggregateBy (AggregateBy.java:removeEldestEntry(328)) - mem on flush (mb), free: 187, total: 498, max: 672
2015-02-12 16:12:49,513 INFO assembly.AggregateBy (AggregateBy.java:removeEldestEntry(333)) - total mem is 74% of max mem, to better utilize unused memory consider increasing current LRU threshold with system property "cascading.aggregateby.threshold"
2015-02-12 16:12:49,513 INFO assembly.AggregateBy (AggregateBy.java:removeEldestEntry(327)) - flushed keys num times: 7, with threshold: 2
2015-02-12 16:12:49,513 INFO assembly.AggregateBy (AggregateBy.java:removeEldestEntry(328)) - mem on flush (mb), free: 187, total: 498, max: 672
2015-02-12 16:12:49,513 INFO assembly.AggregateBy (AggregateBy.java:removeEldestEntry(333)) - total mem is 74% of max mem, to better utilize unused memory consider increasing current LRU threshold with system property "cascading.aggregateby.threshold"
2015-02-12 16:12:49,514 INFO mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) -
2015-02-12 16:12:49,514 INFO mapred.MapTask (MapTask.java:flush(1457)) - Starting flush of map output
2015-02-12 16:12:49,514 INFO mapred.MapTask (MapTask.java:flush(1475)) - Spilling map output
2015-02-12 16:12:49,515 INFO mapred.MapTask (MapTask.java:flush(1476)) - bufstart = 0; bufend = 135; bufvoid = 104857600
2015-02-12 16:12:49,515 INFO mapred.MapTask (MapTask.java:flush(1478)) - kvstart = 26214396(104857584); kvend = 26214364(104857456); length = 33/6553600
2015-02-12 16:12:49,517 INFO mapred.MapTask (MapTask.java:sortAndSpill(1660)) - Finished spill 0
2015-02-12 16:12:49,518 INFO mapred.Task (Task.java:done(1001)) - Task:attempt_local1907556576_0014_m_000000_0 is done. And is in the process of committing
2015-02-12 16:12:49,520 INFO mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) - file:/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/classes/test/data/lhs.txt:0+52
2015-02-12 16:12:49,520 INFO mapred.Task (Task.java:sendDone(1121)) - Task 'attempt_local1907556576_0014_m_000000_0' done.
2015-02-12 16:12:49,520 INFO mapred.LocalJobRunner (LocalJobRunner.java:run(249)) - Finishing task: attempt_local1907556576_0014_m_000000_0
2015-02-12 16:12:49,520 INFO mapred.LocalJobRunner (LocalJobRunner.java:runTasks(456)) - map task executor complete.
2015-02-12 16:12:49,521 INFO mapred.LocalJobRunner (LocalJobRunner.java:runTasks(448)) - Waiting for reduce tasks
2015-02-12 16:12:49,521 INFO mapred.LocalJobRunner (LocalJobRunner.java:run(302)) - Starting task: attempt_local1907556576_0014_r_000000_0
2015-02-12 16:12:49,522 INFO util.ProcfsBasedProcessTree (ProcfsBasedProcessTree.java:isAvailable(181)) - ProcfsBasedProcessTree currently is supported only on Linux.
2015-02-12 16:12:49,523 INFO mapred.Task (Task.java:initialize(587)) - Using ResourceCalculatorProcessTree : null
2015-02-12 16:12:49,523 INFO mapred.ReduceTask (ReduceTask.java:run(362)) - Using ShuffleConsumerPlugin: org.apache.hadoop.mapreduce.task.reduce.Shuffle@45d2d12f
2015-02-12 16:12:49,523 INFO reduce.MergeManagerImpl (MergeManagerImpl.java:<init>(195)) - MergerManager: memoryLimit=493250144, maxSingleShuffleLimit=123312536, mergeThreshold=325545120, ioSortFactor=10, memToMemMergeOutputsThreshold=10
2015-02-12 16:12:49,524 INFO reduce.EventFetcher (EventFetcher.java:run(61)) - attempt_local1907556576_0014_r_000000_0 Thread started: EventFetcher for fetching Map Completion Events
2015-02-12 16:12:49,525 INFO reduce.LocalFetcher (LocalFetcher.java:copyMapOutput(141)) - localfetcher#12 about to shuffle output of map attempt_local1907556576_0014_m_000000_0 decomp: 155 len: 159 to MEMORY
2015-02-12 16:12:49,526 INFO reduce.InMemoryMapOutput (InMemoryMapOutput.java:shuffle(100)) - Read 155 bytes from map-output for attempt_local1907556576_0014_m_000000_0
2015-02-12 16:12:49,526 INFO reduce.MergeManagerImpl (MergeManagerImpl.java:closeInMemoryFile(313)) - closeInMemoryFile -> map-output of size: 155, inMemoryMapOutputs.size() -> 1, commitMemory -> 0, usedMemory ->155
2015-02-12 16:12:49,526 INFO reduce.EventFetcher (EventFetcher.java:run(76)) - EventFetcher is interrupted.. Returning
2015-02-12 16:12:49,527 INFO mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) - 1 / 1 copied.
2015-02-12 16:12:49,527 INFO reduce.MergeManagerImpl (MergeManagerImpl.java:finalMerge(673)) - finalMerge called with 1 in-memory map-outputs and 0 on-disk map-outputs
2015-02-12 16:12:49,529 INFO mapred.Merger (Merger.java:merge(606)) - Merging 1 sorted segments
2015-02-12 16:12:49,529 INFO mapred.Merger (Merger.java:merge(705)) - Down to the last merge-pass, with 1 segments left of total size: 146 bytes
2015-02-12 16:12:49,530 INFO reduce.MergeManagerImpl (MergeManagerImpl.java:finalMerge(750)) - Merged 1 segments, 155 bytes to disk to satisfy reduce memory limit
2015-02-12 16:12:49,531 INFO reduce.MergeManagerImpl (MergeManagerImpl.java:finalMerge(780)) - Merging 1 files, 159 bytes from disk
2015-02-12 16:12:49,531 INFO reduce.MergeManagerImpl (MergeManagerImpl.java:finalMerge(795)) - Merging 0 segments, 0 bytes from memory into reduce
2015-02-12 16:12:49,531 INFO mapred.Merger (Merger.java:merge(606)) - Merging 1 sorted segments
2015-02-12 16:12:49,531 INFO mapred.Merger (Merger.java:merge(705)) - Down to the last merge-pass, with 1 segments left of total size: 146 bytes
2015-02-12 16:12:49,532 INFO mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) - 1 / 1 copied.
2015-02-12 16:12:49,532 INFO hadoop.FlowReducer (FlowReducer.java:configure(78)) - cascading version: 2.6.3
2015-02-12 16:12:49,532 INFO hadoop.FlowReducer (FlowReducer.java:configure(79)) - child jvm opts: -Xmx200m
2015-02-12 16:12:49,539 INFO hadoop.FlowReducer (FlowReducer.java:configure(97)) - sourcing from: GroupBy(min)[by:[{1}:'char']]
2015-02-12 16:12:49,539 INFO hadoop.FlowReducer (FlowReducer.java:configure(100)) - sinking to: Hfs["TextDelimited[['char', 'min']]"]["/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/test/output/hadoop2-mr1/assemblyhelpersplatform/minby"]
2015-02-12 16:12:49,549 INFO mapred.Task (Task.java:done(1001)) - Task:attempt_local1907556576_0014_r_000000_0 is done. And is in the process of committing
2015-02-12 16:12:49,550 INFO mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) - 1 / 1 copied.
2015-02-12 16:12:49,551 INFO mapred.Task (Task.java:commit(1162)) - Task attempt_local1907556576_0014_r_000000_0 is allowed to commit now
2015-02-12 16:12:49,551 INFO output.FileOutputCommitter (FileOutputCommitter.java:commitTask(439)) - Saved output of task 'attempt_local1907556576_0014_r_000000_0' to file:/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/test/output/hadoop2-mr1/assemblyhelpersplatform/minby/_temporary/0/task_local1907556576_0014_r_000000
2015-02-12 16:12:49,552 INFO mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) - reduce > reduce
2015-02-12 16:12:49,552 INFO mapred.Task (Task.java:sendDone(1121)) - Task 'attempt_local1907556576_0014_r_000000_0' done.
2015-02-12 16:12:49,553 INFO mapred.LocalJobRunner (LocalJobRunner.java:run(325)) - Finishing task: attempt_local1907556576_0014_r_000000_0
2015-02-12 16:12:49,553 INFO mapred.LocalJobRunner (LocalJobRunner.java:runTasks(456)) - reduce task executor complete.
2015-02-12 16:12:49,573 INFO util.Hadoop18TapUtil (Hadoop18TapUtil.java:cleanTempPath(219)) - deleting temp path /Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/test/output/hadoop2-mr1/assemblyhelpersplatform/minby/_temporary
2015-02-12 16:12:49,605 INFO mapred.FileInputFormat (FileInputFormat.java:listStatus(247)) - Total input paths to process : 1
2015-02-12 16:12:49,637 INFO mapred.FileInputFormat (FileInputFormat.java:listStatus(247)) - Total input paths to process : 1
2015-02-12 16:12:49,638 INFO cascading.PlatformTestCase (PlatformTestCase.java:tearDown(209)) - copying to local /Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/test/output//hadoop2-mr1/assemblyhelpersplatform/minby
2015-02-12 16:12:49,655 INFO util.HadoopUtil (HadoopUtil.java:findMainClass(336)) - using default application jar, may cause class not found exceptions on the cluster
2015-02-12 16:12:49,655 INFO planner.HadoopPlanner (HadoopPlanner.java:initialize(225)) - using application jar: /Users/soberoi/.gradle/caches/modules-2/files-2.1/cascading/cascading-hadoop2-mr1/2.6.3/9ae016e2b8d246b9c1de9168e715d9f1b971aa4a/cascading-hadoop2-mr1-2.6.3.jar
2015-02-12 16:12:49,659 INFO Configuration.deprecation (Configuration.java:warnOnceIfDeprecated(1022)) - mapred.used.genericoptionsparser is deprecated. Instead, use mapreduce.client.genericoptionsparser.used
2015-02-12 16:12:49,661 INFO Configuration.deprecation (Configuration.java:warnOnceIfDeprecated(1022)) - mapred.job.tracker is deprecated. Instead, use mapreduce.jobtracker.address
2015-02-12 16:12:49,661 INFO Configuration.deprecation (Configuration.java:warnOnceIfDeprecated(1022)) - mapred.output.compress is deprecated. Instead, use mapreduce.output.fileoutputformat.compress
2015-02-12 16:12:49,666 INFO flow.Flow (BaseFlow.java:logInfo(1378)) - [sum] starting
2015-02-12 16:12:49,667 INFO flow.Flow (BaseFlow.java:logInfo(1378)) - [sum] source: Hfs["TextDelimited[['num', 'char']]"]["/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/classes/test/data/lhs.txt"]
2015-02-12 16:12:49,667 INFO flow.Flow (BaseFlow.java:logInfo(1378)) - [sum] sink: Hfs["TextDelimited[['char', 'sum']]"]["/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/test/output/hadoop2-mr1/assemblyhelpersplatform/sum"]
2015-02-12 16:12:49,667 INFO flow.Flow (BaseFlow.java:logInfo(1378)) - [sum] parallel execution is enabled: false
2015-02-12 16:12:49,667 INFO flow.Flow (BaseFlow.java:logInfo(1378)) - [sum] starting jobs: 1
2015-02-12 16:12:49,667 INFO flow.Flow (BaseFlow.java:logInfo(1378)) - [sum] allocating threads: 1
2015-02-12 16:12:49,668 INFO flow.FlowStep (BaseFlowStep.java:logInfo(834)) - [sum] starting step: (1/1) ...semblyhelpersplatform/sum
2015-02-12 16:12:49,668 INFO jvm.JvmMetrics (JvmMetrics.java:init(71)) - Cannot initialize JVM Metrics with processName=JobTracker, sessionId= - already initialized
2015-02-12 16:12:49,669 INFO jvm.JvmMetrics (JvmMetrics.java:init(71)) - Cannot initialize JVM Metrics with processName=JobTracker, sessionId= - already initialized
2015-02-12 16:12:49,689 INFO mapred.FileInputFormat (FileInputFormat.java:listStatus(247)) - Total input paths to process : 1
2015-02-12 16:12:49,705 INFO mapreduce.JobSubmitter (JobSubmitter.java:submitJobInternal(492)) - number of splits:1
2015-02-12 16:12:49,716 INFO mapreduce.JobSubmitter (JobSubmitter.java:printTokens(575)) - Submitting tokens for job: job_local1346937241_0015
2015-02-12 16:12:49,726 WARN conf.Configuration (Configuration.java:loadProperty(2510)) - file:/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/tmp/cascading/staging/soberoi1346937241/.staging/job_local1346937241_0015/job.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.retry.interval; Ignoring.
2015-02-12 16:12:49,727 WARN conf.Configuration (Configuration.java:loadProperty(2510)) - file:/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/tmp/cascading/staging/soberoi1346937241/.staging/job_local1346937241_0015/job.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.attempts; Ignoring.
2015-02-12 16:12:49,757 WARN conf.Configuration (Configuration.java:loadProperty(2510)) - file:/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/test/tmp/mapred/local/localRunner/soberoi/job_local1346937241_0015/job_local1346937241_0015.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.retry.interval; Ignoring.
2015-02-12 16:12:49,758 WARN conf.Configuration (Configuration.java:loadProperty(2510)) - file:/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/test/tmp/mapred/local/localRunner/soberoi/job_local1346937241_0015/job_local1346937241_0015.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.attempts; Ignoring.
2015-02-12 16:12:49,759 INFO mapreduce.Job (Job.java:submit(1299)) - The url to track the job: http://localhost:8080/
2015-02-12 16:12:49,759 INFO mapred.LocalJobRunner (LocalJobRunner.java:createOutputCommitter(471)) - OutputCommitter set in config null
2015-02-12 16:12:49,759 INFO flow.FlowStep (BaseFlowStep.java:logInfo(834)) - [sum] submitted hadoop job: job_local1346937241_0015
2015-02-12 16:12:49,759 INFO mapred.LocalJobRunner (LocalJobRunner.java:createOutputCommitter(489)) - OutputCommitter is org.apache.hadoop.mapred.FileOutputCommitter
2015-02-12 16:12:49,759 INFO flow.FlowStep (BaseFlowStep.java:logInfo(834)) - [sum] tracking url: http://localhost:8080/
2015-02-12 16:12:49,761 INFO mapred.LocalJobRunner (LocalJobRunner.java:runTasks(448)) - Waiting for map tasks
2015-02-12 16:12:49,761 INFO mapred.LocalJobRunner (LocalJobRunner.java:run(224)) - Starting task: attempt_local1346937241_0015_m_000000_0
2015-02-12 16:12:49,762 INFO util.ProcfsBasedProcessTree (ProcfsBasedProcessTree.java:isAvailable(181)) - ProcfsBasedProcessTree currently is supported only on Linux.
2015-02-12 16:12:49,762 INFO mapred.Task (Task.java:initialize(587)) - Using ResourceCalculatorProcessTree : null
2015-02-12 16:12:49,763 INFO io.MultiInputSplit (MultiInputSplit.java:readFields(161)) - current split input path: file:/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/classes/test/data/lhs.txt
2015-02-12 16:12:49,763 INFO mapred.MapTask (MapTask.java:updateJobWithSplit(482)) - Processing split: cascading.tap.hadoop.io.MultiInputSplit@3e708bac
2015-02-12 16:12:49,764 INFO mapred.MapTask (MapTask.java:runOldMapper(436)) - numReduceTasks: 1
2015-02-12 16:12:49,780 INFO mapred.MapTask (MapTask.java:setEquator(1202)) - (EQUATOR) 0 kvi 26214396(104857584)
2015-02-12 16:12:49,780 INFO mapred.MapTask (MapTask.java:init(995)) - mapreduce.task.io.sort.mb: 100
2015-02-12 16:12:49,780 INFO mapred.MapTask (MapTask.java:init(996)) - soft limit at 83886080
2015-02-12 16:12:49,780 INFO mapred.MapTask (MapTask.java:init(997)) - bufstart = 0; bufvoid = 104857600
2015-02-12 16:12:49,780 INFO mapred.MapTask (MapTask.java:init(998)) - kvstart = 26214396; length = 6553600
2015-02-12 16:12:49,781 INFO mapred.MapTask (MapTask.java:createSortingCollector(402)) - Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer
2015-02-12 16:12:49,781 INFO hadoop.FlowMapper (FlowMapper.java:configure(71)) - cascading version: 2.6.3
2015-02-12 16:12:49,781 INFO hadoop.FlowMapper (FlowMapper.java:configure(72)) - child jvm opts: -Xmx200m
2015-02-12 16:12:49,786 INFO hadoop.FlowMapper (FlowMapper.java:configure(87)) - sourcing from: Hfs["TextDelimited[['num', 'char']]"]["/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/classes/test/data/lhs.txt"]
2015-02-12 16:12:49,786 INFO hadoop.FlowMapper (FlowMapper.java:configure(90)) - sinking to: GroupBy(sum)[by:[{1}:'char']]
2015-02-12 16:12:49,786 INFO assembly.AggregateBy (AggregateBy.java:prepare(267)) - using threshold value: 2
2015-02-12 16:12:49,787 INFO assembly.AggregateBy (AggregateBy.java:removeEldestEntry(327)) - flushed keys num times: 1, with threshold: 2
2015-02-12 16:12:49,787 INFO assembly.AggregateBy (AggregateBy.java:removeEldestEntry(328)) - mem on flush (mb), free: 197, total: 509, max: 672
2015-02-12 16:12:49,787 INFO assembly.AggregateBy (AggregateBy.java:removeEldestEntry(333)) - total mem is 75% of max mem, to better utilize unused memory consider increasing current LRU threshold with system property "cascading.aggregateby.threshold"
2015-02-12 16:12:49,788 INFO assembly.AggregateBy (AggregateBy.java:removeEldestEntry(327)) - flushed keys num times: 3, with threshold: 2
2015-02-12 16:12:49,788 INFO assembly.AggregateBy (AggregateBy.java:removeEldestEntry(328)) - mem on flush (mb), free: 197, total: 509, max: 672
2015-02-12 16:12:49,788 INFO assembly.AggregateBy (AggregateBy.java:removeEldestEntry(333)) - total mem is 75% of max mem, to better utilize unused memory consider increasing current LRU threshold with system property "cascading.aggregateby.threshold"
2015-02-12 16:12:49,788 INFO assembly.AggregateBy (AggregateBy.java:removeEldestEntry(327)) - flushed keys num times: 5, with threshold: 2
2015-02-12 16:12:49,788 INFO assembly.AggregateBy (AggregateBy.java:removeEldestEntry(328)) - mem on flush (mb), free: 197, total: 509, max: 672
2015-02-12 16:12:49,788 INFO assembly.AggregateBy (AggregateBy.java:removeEldestEntry(333)) - total mem is 75% of max mem, to better utilize unused memory consider increasing current LRU threshold with system property "cascading.aggregateby.threshold"
2015-02-12 16:12:49,789 INFO assembly.AggregateBy (AggregateBy.java:removeEldestEntry(327)) - flushed keys num times: 7, with threshold: 2
2015-02-12 16:12:49,789 INFO assembly.AggregateBy (AggregateBy.java:removeEldestEntry(328)) - mem on flush (mb), free: 197, total: 509, max: 672
2015-02-12 16:12:49,789 INFO assembly.AggregateBy (AggregateBy.java:removeEldestEntry(333)) - total mem is 75% of max mem, to better utilize unused memory consider increasing current LRU threshold with system property "cascading.aggregateby.threshold"
2015-02-12 16:12:49,790 INFO mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) -
2015-02-12 16:12:49,790 INFO mapred.MapTask (MapTask.java:flush(1457)) - Starting flush of map output
2015-02-12 16:12:49,790 INFO mapred.MapTask (MapTask.java:flush(1475)) - Spilling map output
2015-02-12 16:12:49,790 INFO mapred.MapTask (MapTask.java:flush(1476)) - bufstart = 0; bufend = 99; bufvoid = 104857600
2015-02-12 16:12:49,790 INFO mapred.MapTask (MapTask.java:flush(1478)) - kvstart = 26214396(104857584); kvend = 26214364(104857456); length = 33/6553600
2015-02-12 16:12:49,792 INFO mapred.MapTask (MapTask.java:sortAndSpill(1660)) - Finished spill 0
2015-02-12 16:12:49,793 INFO mapred.Task (Task.java:done(1001)) - Task:attempt_local1346937241_0015_m_000000_0 is done. And is in the process of committing
2015-02-12 16:12:49,794 INFO mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) - file:/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/classes/test/data/lhs.txt:0+52
2015-02-12 16:12:49,794 INFO mapred.Task (Task.java:sendDone(1121)) - Task 'attempt_local1346937241_0015_m_000000_0' done.
2015-02-12 16:12:49,795 INFO mapred.LocalJobRunner (LocalJobRunner.java:run(249)) - Finishing task: attempt_local1346937241_0015_m_000000_0
2015-02-12 16:12:49,795 INFO mapred.LocalJobRunner (LocalJobRunner.java:runTasks(456)) - map task executor complete.
2015-02-12 16:12:49,795 INFO mapred.LocalJobRunner (LocalJobRunner.java:runTasks(448)) - Waiting for reduce tasks
2015-02-12 16:12:49,795 INFO mapred.LocalJobRunner (LocalJobRunner.java:run(302)) - Starting task: attempt_local1346937241_0015_r_000000_0
2015-02-12 16:12:49,796 INFO util.ProcfsBasedProcessTree (ProcfsBasedProcessTree.java:isAvailable(181)) - ProcfsBasedProcessTree currently is supported only on Linux.
2015-02-12 16:12:49,796 INFO mapred.Task (Task.java:initialize(587)) - Using ResourceCalculatorProcessTree : null
2015-02-12 16:12:49,797 INFO mapred.ReduceTask (ReduceTask.java:run(362)) - Using ShuffleConsumerPlugin: org.apache.hadoop.mapreduce.task.reduce.Shuffle@50e132fe
2015-02-12 16:12:49,797 INFO reduce.MergeManagerImpl (MergeManagerImpl.java:<init>(195)) - MergerManager: memoryLimit=493250144, maxSingleShuffleLimit=123312536, mergeThreshold=325545120, ioSortFactor=10, memToMemMergeOutputsThreshold=10
2015-02-12 16:12:49,798 INFO reduce.EventFetcher (EventFetcher.java:run(61)) - attempt_local1346937241_0015_r_000000_0 Thread started: EventFetcher for fetching Map Completion Events
2015-02-12 16:12:49,798 INFO reduce.LocalFetcher (LocalFetcher.java:copyMapOutput(141)) - localfetcher#13 about to shuffle output of map attempt_local1346937241_0015_m_000000_0 decomp: 119 len: 123 to MEMORY
2015-02-12 16:12:49,799 INFO reduce.InMemoryMapOutput (InMemoryMapOutput.java:shuffle(100)) - Read 119 bytes from map-output for attempt_local1346937241_0015_m_000000_0
2015-02-12 16:12:49,799 INFO reduce.MergeManagerImpl (MergeManagerImpl.java:closeInMemoryFile(313)) - closeInMemoryFile -> map-output of size: 119, inMemoryMapOutputs.size() -> 1, commitMemory -> 0, usedMemory ->119
2015-02-12 16:12:49,799 INFO reduce.EventFetcher (EventFetcher.java:run(76)) - EventFetcher is interrupted.. Returning
2015-02-12 16:12:49,800 INFO mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) - 1 / 1 copied.
2015-02-12 16:12:49,800 INFO reduce.MergeManagerImpl (MergeManagerImpl.java:finalMerge(673)) - finalMerge called with 1 in-memory map-outputs and 0 on-disk map-outputs
2015-02-12 16:12:49,801 INFO mapred.Merger (Merger.java:merge(606)) - Merging 1 sorted segments
2015-02-12 16:12:49,801 INFO mapred.Merger (Merger.java:merge(705)) - Down to the last merge-pass, with 1 segments left of total size: 110 bytes
2015-02-12 16:12:49,802 INFO reduce.MergeManagerImpl (MergeManagerImpl.java:finalMerge(750)) - Merged 1 segments, 119 bytes to disk to satisfy reduce memory limit
2015-02-12 16:12:49,802 INFO reduce.MergeManagerImpl (MergeManagerImpl.java:finalMerge(780)) - Merging 1 files, 123 bytes from disk
2015-02-12 16:12:49,803 INFO reduce.MergeManagerImpl (MergeManagerImpl.java:finalMerge(795)) - Merging 0 segments, 0 bytes from memory into reduce
2015-02-12 16:12:49,803 INFO mapred.Merger (Merger.java:merge(606)) - Merging 1 sorted segments
2015-02-12 16:12:49,803 INFO mapred.Merger (Merger.java:merge(705)) - Down to the last merge-pass, with 1 segments left of total size: 110 bytes
2015-02-12 16:12:49,803 INFO mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) - 1 / 1 copied.
2015-02-12 16:12:49,804 INFO hadoop.FlowReducer (FlowReducer.java:configure(78)) - cascading version: 2.6.3
2015-02-12 16:12:49,804 INFO hadoop.FlowReducer (FlowReducer.java:configure(79)) - child jvm opts: -Xmx200m
2015-02-12 16:12:49,809 INFO hadoop.FlowReducer (FlowReducer.java:configure(97)) - sourcing from: GroupBy(sum)[by:[{1}:'char']]
2015-02-12 16:12:49,810 INFO hadoop.FlowReducer (FlowReducer.java:configure(100)) - sinking to: Hfs["TextDelimited[['char', 'sum']]"]["/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/test/output/hadoop2-mr1/assemblyhelpersplatform/sum"]
2015-02-12 16:12:49,818 INFO mapred.Task (Task.java:done(1001)) - Task:attempt_local1346937241_0015_r_000000_0 is done. And is in the process of committing
2015-02-12 16:12:49,819 INFO mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) - 1 / 1 copied.
2015-02-12 16:12:49,819 INFO mapred.Task (Task.java:commit(1162)) - Task attempt_local1346937241_0015_r_000000_0 is allowed to commit now
2015-02-12 16:12:49,820 INFO output.FileOutputCommitter (FileOutputCommitter.java:commitTask(439)) - Saved output of task 'attempt_local1346937241_0015_r_000000_0' to file:/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/test/output/hadoop2-mr1/assemblyhelpersplatform/sum/_temporary/0/task_local1346937241_0015_r_000000
2015-02-12 16:12:49,820 INFO mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) - reduce > reduce
2015-02-12 16:12:49,821 INFO mapred.Task (Task.java:sendDone(1121)) - Task 'attempt_local1346937241_0015_r_000000_0' done.
2015-02-12 16:12:49,821 INFO mapred.LocalJobRunner (LocalJobRunner.java:run(325)) - Finishing task: attempt_local1346937241_0015_r_000000_0
2015-02-12 16:12:49,821 INFO mapred.LocalJobRunner (LocalJobRunner.java:runTasks(456)) - reduce task executor complete.
2015-02-12 16:12:49,841 INFO util.Hadoop18TapUtil (Hadoop18TapUtil.java:cleanTempPath(219)) - deleting temp path /Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/test/output/hadoop2-mr1/assemblyhelpersplatform/sum/_temporary
2015-02-12 16:12:49,861 INFO mapred.FileInputFormat (FileInputFormat.java:listStatus(247)) - Total input paths to process : 1
2015-02-12 16:12:49,883 INFO mapred.FileInputFormat (FileInputFormat.java:listStatus(247)) - Total input paths to process : 1
2015-02-12 16:12:49,884 INFO cascading.PlatformTestCase (PlatformTestCase.java:tearDown(209)) - copying to local /Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/test/output//hadoop2-mr1/assemblyhelpersplatform/sum
2015-02-12 16:12:49,897 INFO util.HadoopUtil (HadoopUtil.java:findMainClass(336)) - using default application jar, may cause class not found exceptions on the cluster
2015-02-12 16:12:49,897 INFO planner.HadoopPlanner (HadoopPlanner.java:initialize(225)) - using application jar: /Users/soberoi/.gradle/caches/modules-2/files-2.1/cascading/cascading-hadoop2-mr1/2.6.3/9ae016e2b8d246b9c1de9168e715d9f1b971aa4a/cascading-hadoop2-mr1-2.6.3.jar
2015-02-12 16:12:49,903 INFO Configuration.deprecation (Configuration.java:warnOnceIfDeprecated(1022)) - mapred.used.genericoptionsparser is deprecated. Instead, use mapreduce.client.genericoptionsparser.used
2015-02-12 16:12:49,905 INFO Configuration.deprecation (Configuration.java:warnOnceIfDeprecated(1022)) - mapred.job.tracker is deprecated. Instead, use mapreduce.jobtracker.address
2015-02-12 16:12:49,905 INFO Configuration.deprecation (Configuration.java:warnOnceIfDeprecated(1022)) - mapred.output.compress is deprecated. Instead, use mapreduce.output.fileoutputformat.compress
2015-02-12 16:12:49,910 INFO flow.Flow (BaseFlow.java:logInfo(1378)) - [sum-lhs+sum-rhs] starting
2015-02-12 16:12:49,910 INFO flow.Flow (BaseFlow.java:logInfo(1378)) - [sum-lhs+sum-rhs] source: Hfs["TextDelimited[['num', 'char']]"]["/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/classes/test/data/rhs.txt"]
2015-02-12 16:12:49,911 INFO flow.Flow (BaseFlow.java:logInfo(1378)) - [sum-lhs+sum-rhs] source: Hfs["TextDelimited[['num', 'char']]"]["/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/classes/test/data/lhs.txt"]
2015-02-12 16:12:49,911 INFO flow.Flow (BaseFlow.java:logInfo(1378)) - [sum-lhs+sum-rhs] sink: Hfs["TextDelimited[['char', 'sum']]"]["/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/test/output/hadoop2-mr1/assemblyhelpersplatform/mergesum"]
2015-02-12 16:12:49,911 INFO flow.Flow (BaseFlow.java:logInfo(1378)) - [sum-lhs+sum-rhs] parallel execution is enabled: false
2015-02-12 16:12:49,911 INFO flow.Flow (BaseFlow.java:logInfo(1378)) - [sum-lhs+sum-rhs] starting jobs: 1
2015-02-12 16:12:49,911 INFO flow.Flow (BaseFlow.java:logInfo(1378)) - [sum-lhs+sum-rhs] allocating threads: 1
2015-02-12 16:12:49,911 INFO flow.FlowStep (BaseFlowStep.java:logInfo(834)) - [sum-lhs+sum-rhs] starting step: (1/1) ...yhelpersplatform/mergesum
2015-02-12 16:12:49,912 INFO jvm.JvmMetrics (JvmMetrics.java:init(71)) - Cannot initialize JVM Metrics with processName=JobTracker, sessionId= - already initialized
2015-02-12 16:12:49,913 INFO jvm.JvmMetrics (JvmMetrics.java:init(71)) - Cannot initialize JVM Metrics with processName=JobTracker, sessionId= - already initialized
2015-02-12 16:12:49,931 INFO mapred.FileInputFormat (FileInputFormat.java:listStatus(247)) - Total input paths to process : 1
2015-02-12 16:12:49,932 INFO mapred.FileInputFormat (FileInputFormat.java:listStatus(247)) - Total input paths to process : 1
2015-02-12 16:12:49,953 INFO mapreduce.JobSubmitter (JobSubmitter.java:submitJobInternal(492)) - number of splits:2
2015-02-12 16:12:49,964 INFO mapreduce.JobSubmitter (JobSubmitter.java:printTokens(575)) - Submitting tokens for job: job_local1316284914_0016
2015-02-12 16:12:49,976 WARN conf.Configuration (Configuration.java:loadProperty(2510)) - file:/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/tmp/cascading/staging/soberoi1316284914/.staging/job_local1316284914_0016/job.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.retry.interval; Ignoring.
2015-02-12 16:12:49,977 WARN conf.Configuration (Configuration.java:loadProperty(2510)) - file:/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/tmp/cascading/staging/soberoi1316284914/.staging/job_local1316284914_0016/job.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.attempts; Ignoring.
2015-02-12 16:12:50,008 WARN conf.Configuration (Configuration.java:loadProperty(2510)) - file:/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/test/tmp/mapred/local/localRunner/soberoi/job_local1316284914_0016/job_local1316284914_0016.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.retry.interval; Ignoring.
2015-02-12 16:12:50,009 WARN conf.Configuration (Configuration.java:loadProperty(2510)) - file:/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/test/tmp/mapred/local/localRunner/soberoi/job_local1316284914_0016/job_local1316284914_0016.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.attempts; Ignoring.
2015-02-12 16:12:50,009 INFO mapreduce.Job (Job.java:submit(1299)) - The url to track the job: http://localhost:8080/
2015-02-12 16:12:50,009 INFO mapred.LocalJobRunner (LocalJobRunner.java:createOutputCommitter(471)) - OutputCommitter set in config null
2015-02-12 16:12:50,010 INFO mapred.LocalJobRunner (LocalJobRunner.java:createOutputCommitter(489)) - OutputCommitter is org.apache.hadoop.mapred.FileOutputCommitter
2015-02-12 16:12:50,009 INFO flow.FlowStep (BaseFlowStep.java:logInfo(834)) - [sum-lhs+sum-rhs] submitted hadoop job: job_local1316284914_0016
2015-02-12 16:12:50,010 INFO flow.FlowStep (BaseFlowStep.java:logInfo(834)) - [sum-lhs+sum-rhs] tracking url: http://localhost:8080/
2015-02-12 16:12:50,011 INFO mapred.LocalJobRunner (LocalJobRunner.java:runTasks(448)) - Waiting for map tasks
2015-02-12 16:12:50,012 INFO mapred.LocalJobRunner (LocalJobRunner.java:run(224)) - Starting task: attempt_local1316284914_0016_m_000000_0
2015-02-12 16:12:50,013 INFO util.ProcfsBasedProcessTree (ProcfsBasedProcessTree.java:isAvailable(181)) - ProcfsBasedProcessTree currently is supported only on Linux.
2015-02-12 16:12:50,013 INFO mapred.Task (Task.java:initialize(587)) - Using ResourceCalculatorProcessTree : null
2015-02-12 16:12:50,013 INFO io.MultiInputSplit (MultiInputSplit.java:readFields(161)) - current split input path: file:/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/classes/test/data/lhs.txt
2015-02-12 16:12:50,014 INFO mapred.MapTask (MapTask.java:updateJobWithSplit(482)) - Processing split: cascading.tap.hadoop.io.MultiInputSplit@3ae24197
2015-02-12 16:12:50,014 INFO mapred.MapTask (MapTask.java:runOldMapper(436)) - numReduceTasks: 1
2015-02-12 16:12:50,029 INFO mapred.MapTask (MapTask.java:setEquator(1202)) - (EQUATOR) 0 kvi 26214396(104857584)
2015-02-12 16:12:50,030 INFO mapred.MapTask (MapTask.java:init(995)) - mapreduce.task.io.sort.mb: 100
2015-02-12 16:12:50,030 INFO mapred.MapTask (MapTask.java:init(996)) - soft limit at 83886080
2015-02-12 16:12:50,030 INFO mapred.MapTask (MapTask.java:init(997)) - bufstart = 0; bufvoid = 104857600
2015-02-12 16:12:50,030 INFO mapred.MapTask (MapTask.java:init(998)) - kvstart = 26214396; length = 6553600
2015-02-12 16:12:50,031 INFO mapred.MapTask (MapTask.java:createSortingCollector(402)) - Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer
2015-02-12 16:12:50,031 INFO hadoop.FlowMapper (FlowMapper.java:configure(71)) - cascading version: 2.6.3
2015-02-12 16:12:50,031 INFO hadoop.FlowMapper (FlowMapper.java:configure(72)) - child jvm opts: -Xmx200m
2015-02-12 16:12:50,038 INFO hadoop.FlowMapper (FlowMapper.java:configure(87)) - sourcing from: Hfs["TextDelimited[['num', 'char']]"]["/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/classes/test/data/lhs.txt"]
2015-02-12 16:12:50,038 INFO hadoop.FlowMapper (FlowMapper.java:configure(90)) - sinking to: GroupBy(sum-lhs+sum-rhs)[by: sum-lhs:[{1}:'char'] sum-rhs:[{1}:'char']]
2015-02-12 16:12:50,038 INFO assembly.AggregateBy (AggregateBy.java:prepare(267)) - using threshold value: 2
2015-02-12 16:12:50,039 INFO assembly.AggregateBy (AggregateBy.java:removeEldestEntry(327)) - flushed keys num times: 1, with threshold: 2
2015-02-12 16:12:50,039 INFO assembly.AggregateBy (AggregateBy.java:removeEldestEntry(328)) - mem on flush (mb), free: 197, total: 509, max: 672
2015-02-12 16:12:50,039 INFO assembly.AggregateBy (AggregateBy.java:removeEldestEntry(333)) - total mem is 75% of max mem, to better utilize unused memory consider increasing current LRU threshold with system property "cascading.aggregateby.threshold"
2015-02-12 16:12:50,040 INFO assembly.AggregateBy (AggregateBy.java:removeEldestEntry(327)) - flushed keys num times: 3, with threshold: 2
2015-02-12 16:12:50,040 INFO assembly.AggregateBy (AggregateBy.java:removeEldestEntry(328)) - mem on flush (mb), free: 197, total: 509, max: 672
2015-02-12 16:12:50,040 INFO assembly.AggregateBy (AggregateBy.java:removeEldestEntry(333)) - total mem is 75% of max mem, to better utilize unused memory consider increasing current LRU threshold with system property "cascading.aggregateby.threshold"
2015-02-12 16:12:50,040 INFO assembly.AggregateBy (AggregateBy.java:removeEldestEntry(327)) - flushed keys num times: 5, with threshold: 2
2015-02-12 16:12:50,040 INFO assembly.AggregateBy (AggregateBy.java:removeEldestEntry(328)) - mem on flush (mb), free: 197, total: 509, max: 672
2015-02-12 16:12:50,041 INFO assembly.AggregateBy (AggregateBy.java:removeEldestEntry(333)) - total mem is 75% of max mem, to better utilize unused memory consider increasing current LRU threshold with system property "cascading.aggregateby.threshold"
2015-02-12 16:12:50,041 INFO assembly.AggregateBy (AggregateBy.java:removeEldestEntry(327)) - flushed keys num times: 7, with threshold: 2
2015-02-12 16:12:50,041 INFO assembly.AggregateBy (AggregateBy.java:removeEldestEntry(328)) - mem on flush (mb), free: 197, total: 509, max: 672
2015-02-12 16:12:50,041 INFO assembly.AggregateBy (AggregateBy.java:removeEldestEntry(333)) - total mem is 75% of max mem, to better utilize unused memory consider increasing current LRU threshold with system property "cascading.aggregateby.threshold"
2015-02-12 16:12:50,042 INFO mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) -
2015-02-12 16:12:50,042 INFO mapred.MapTask (MapTask.java:flush(1457)) - Starting flush of map output
2015-02-12 16:12:50,042 INFO mapred.MapTask (MapTask.java:flush(1475)) - Spilling map output
2015-02-12 16:12:50,042 INFO mapred.MapTask (MapTask.java:flush(1476)) - bufstart = 0; bufend = 99; bufvoid = 104857600
2015-02-12 16:12:50,042 INFO mapred.MapTask (MapTask.java:flush(1478)) - kvstart = 26214396(104857584); kvend = 26214364(104857456); length = 33/6553600
2015-02-12 16:12:50,044 INFO mapred.MapTask (MapTask.java:sortAndSpill(1660)) - Finished spill 0
2015-02-12 16:12:50,045 INFO mapred.Task (Task.java:done(1001)) - Task:attempt_local1316284914_0016_m_000000_0 is done. And is in the process of committing
2015-02-12 16:12:50,047 INFO mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) - file:/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/classes/test/data/lhs.txt:0+52
2015-02-12 16:12:50,047 INFO mapred.Task (Task.java:sendDone(1121)) - Task 'attempt_local1316284914_0016_m_000000_0' done.
2015-02-12 16:12:50,047 INFO mapred.LocalJobRunner (LocalJobRunner.java:run(249)) - Finishing task: attempt_local1316284914_0016_m_000000_0
2015-02-12 16:12:50,047 INFO mapred.LocalJobRunner (LocalJobRunner.java:run(224)) - Starting task: attempt_local1316284914_0016_m_000001_0
2015-02-12 16:12:50,048 INFO util.ProcfsBasedProcessTree (ProcfsBasedProcessTree.java:isAvailable(181)) - ProcfsBasedProcessTree currently is supported only on Linux.
2015-02-12 16:12:50,048 INFO mapred.Task (Task.java:initialize(587)) - Using ResourceCalculatorProcessTree : null
2015-02-12 16:12:50,050 INFO io.MultiInputSplit (MultiInputSplit.java:readFields(161)) - current split input path: file:/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/classes/test/data/rhs.txt
2015-02-12 16:12:50,050 INFO mapred.MapTask (MapTask.java:updateJobWithSplit(482)) - Processing split: cascading.tap.hadoop.io.MultiInputSplit@74a32d61
2015-02-12 16:12:50,051 INFO mapred.MapTask (MapTask.java:runOldMapper(436)) - numReduceTasks: 1
2015-02-12 16:12:50,063 INFO mapred.MapTask (MapTask.java:setEquator(1202)) - (EQUATOR) 0 kvi 26214396(104857584)
2015-02-12 16:12:50,063 INFO mapred.MapTask (MapTask.java:init(995)) - mapreduce.task.io.sort.mb: 100
2015-02-12 16:12:50,063 INFO mapred.MapTask (MapTask.java:init(996)) - soft limit at 83886080
2015-02-12 16:12:50,064 INFO mapred.MapTask (MapTask.java:init(997)) - bufstart = 0; bufvoid = 104857600
2015-02-12 16:12:50,064 INFO mapred.MapTask (MapTask.java:init(998)) - kvstart = 26214396; length = 6553600
2015-02-12 16:12:50,064 INFO mapred.MapTask (MapTask.java:createSortingCollector(402)) - Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer
2015-02-12 16:12:50,065 INFO hadoop.FlowMapper (FlowMapper.java:configure(71)) - cascading version: 2.6.3
2015-02-12 16:12:50,065 INFO hadoop.FlowMapper (FlowMapper.java:configure(72)) - child jvm opts: -Xmx200m
2015-02-12 16:12:50,070 INFO hadoop.FlowMapper (FlowMapper.java:configure(87)) - sourcing from: Hfs["TextDelimited[['num', 'char']]"]["/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/classes/test/data/rhs.txt"]
2015-02-12 16:12:50,070 INFO hadoop.FlowMapper (FlowMapper.java:configure(90)) - sinking to: GroupBy(sum-lhs+sum-rhs)[by: sum-lhs:[{1}:'char'] sum-rhs:[{1}:'char']]
2015-02-12 16:12:50,071 INFO assembly.AggregateBy (AggregateBy.java:prepare(267)) - using threshold value: 2
2015-02-12 16:12:50,073 INFO assembly.AggregateBy (AggregateBy.java:removeEldestEntry(327)) - flushed keys num times: 1, with threshold: 2
2015-02-12 16:12:50,073 INFO assembly.AggregateBy (AggregateBy.java:removeEldestEntry(328)) - mem on flush (mb), free: 95, total: 509, max: 672
2015-02-12 16:12:50,073 INFO assembly.AggregateBy (AggregateBy.java:removeEldestEntry(333)) - total mem is 75% of max mem, to better utilize unused memory consider increasing current LRU threshold with system property "cascading.aggregateby.threshold"
2015-02-12 16:12:50,074 INFO assembly.AggregateBy (AggregateBy.java:removeEldestEntry(327)) - flushed keys num times: 3, with threshold: 2
2015-02-12 16:12:50,074 INFO assembly.AggregateBy (AggregateBy.java:removeEldestEntry(328)) - mem on flush (mb), free: 95, total: 509, max: 672
2015-02-12 16:12:50,074 INFO assembly.AggregateBy (AggregateBy.java:removeEldestEntry(333)) - total mem is 75% of max mem, to better utilize unused memory consider increasing current LRU threshold with system property "cascading.aggregateby.threshold"
2015-02-12 16:12:50,075 INFO assembly.AggregateBy (AggregateBy.java:removeEldestEntry(327)) - flushed keys num times: 5, with threshold: 2
2015-02-12 16:12:50,075 INFO assembly.AggregateBy (AggregateBy.java:removeEldestEntry(328)) - mem on flush (mb), free: 95, total: 509, max: 672
2015-02-12 16:12:50,075 INFO assembly.AggregateBy (AggregateBy.java:removeEldestEntry(333)) - total mem is 75% of max mem, to better utilize unused memory consider increasing current LRU threshold with system property "cascading.aggregateby.threshold"
2015-02-12 16:12:50,075 INFO assembly.AggregateBy (AggregateBy.java:removeEldestEntry(327)) - flushed keys num times: 7, with threshold: 2
2015-02-12 16:12:50,075 INFO assembly.AggregateBy (AggregateBy.java:removeEldestEntry(328)) - mem on flush (mb), free: 95, total: 509, max: 672
2015-02-12 16:12:50,076 INFO assembly.AggregateBy (AggregateBy.java:removeEldestEntry(333)) - total mem is 75% of max mem, to better utilize unused memory consider increasing current LRU threshold with system property "cascading.aggregateby.threshold"
2015-02-12 16:12:50,076 INFO mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) -
2015-02-12 16:12:50,076 INFO mapred.MapTask (MapTask.java:flush(1457)) - Starting flush of map output
2015-02-12 16:12:50,076 INFO mapred.MapTask (MapTask.java:flush(1475)) - Spilling map output
2015-02-12 16:12:50,077 INFO mapred.MapTask (MapTask.java:flush(1476)) - bufstart = 0; bufend = 99; bufvoid = 104857600
2015-02-12 16:12:50,077 INFO mapred.MapTask (MapTask.java:flush(1478)) - kvstart = 26214396(104857584); kvend = 26214364(104857456); length = 33/6553600
2015-02-12 16:12:50,078 INFO mapred.MapTask (MapTask.java:sortAndSpill(1660)) - Finished spill 0
2015-02-12 16:12:50,080 INFO mapred.Task (Task.java:done(1001)) - Task:attempt_local1316284914_0016_m_000001_0 is done. And is in the process of committing
2015-02-12 16:12:50,081 INFO mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) - file:/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/classes/test/data/rhs.txt:0+51
2015-02-12 16:12:50,081 INFO mapred.Task (Task.java:sendDone(1121)) - Task 'attempt_local1316284914_0016_m_000001_0' done.
2015-02-12 16:12:50,081 INFO mapred.LocalJobRunner (LocalJobRunner.java:run(249)) - Finishing task: attempt_local1316284914_0016_m_000001_0
2015-02-12 16:12:50,081 INFO mapred.LocalJobRunner (LocalJobRunner.java:runTasks(456)) - map task executor complete.
2015-02-12 16:12:50,082 INFO mapred.LocalJobRunner (LocalJobRunner.java:runTasks(448)) - Waiting for reduce tasks
2015-02-12 16:12:50,082 INFO mapred.LocalJobRunner (LocalJobRunner.java:run(302)) - Starting task: attempt_local1316284914_0016_r_000000_0
2015-02-12 16:12:50,083 INFO util.ProcfsBasedProcessTree (ProcfsBasedProcessTree.java:isAvailable(181)) - ProcfsBasedProcessTree currently is supported only on Linux.
2015-02-12 16:12:50,083 INFO mapred.Task (Task.java:initialize(587)) - Using ResourceCalculatorProcessTree : null
2015-02-12 16:12:50,083 INFO mapred.ReduceTask (ReduceTask.java:run(362)) - Using ShuffleConsumerPlugin: org.apache.hadoop.mapreduce.task.reduce.Shuffle@25719f5c
2015-02-12 16:12:50,083 INFO reduce.MergeManagerImpl (MergeManagerImpl.java:<init>(195)) - MergerManager: memoryLimit=493250144, maxSingleShuffleLimit=123312536, mergeThreshold=325545120, ioSortFactor=10, memToMemMergeOutputsThreshold=10
2015-02-12 16:12:50,084 INFO reduce.EventFetcher (EventFetcher.java:run(61)) - attempt_local1316284914_0016_r_000000_0 Thread started: EventFetcher for fetching Map Completion Events
2015-02-12 16:12:50,084 INFO reduce.LocalFetcher (LocalFetcher.java:copyMapOutput(141)) - localfetcher#14 about to shuffle output of map attempt_local1316284914_0016_m_000001_0 decomp: 119 len: 123 to MEMORY
2015-02-12 16:12:50,085 INFO reduce.InMemoryMapOutput (InMemoryMapOutput.java:shuffle(100)) - Read 119 bytes from map-output for attempt_local1316284914_0016_m_000001_0
2015-02-12 16:12:50,085 INFO reduce.MergeManagerImpl (MergeManagerImpl.java:closeInMemoryFile(313)) - closeInMemoryFile -> map-output of size: 119, inMemoryMapOutputs.size() -> 1, commitMemory -> 0, usedMemory ->119
2015-02-12 16:12:50,086 INFO reduce.LocalFetcher (LocalFetcher.java:copyMapOutput(141)) - localfetcher#14 about to shuffle output of map attempt_local1316284914_0016_m_000000_0 decomp: 119 len: 123 to MEMORY
2015-02-12 16:12:50,086 INFO reduce.InMemoryMapOutput (InMemoryMapOutput.java:shuffle(100)) - Read 119 bytes from map-output for attempt_local1316284914_0016_m_000000_0
2015-02-12 16:12:50,086 INFO reduce.MergeManagerImpl (MergeManagerImpl.java:closeInMemoryFile(313)) - closeInMemoryFile -> map-output of size: 119, inMemoryMapOutputs.size() -> 2, commitMemory -> 119, usedMemory ->238
2015-02-12 16:12:50,087 INFO reduce.EventFetcher (EventFetcher.java:run(76)) - EventFetcher is interrupted.. Returning
2015-02-12 16:12:50,087 INFO mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) - 2 / 2 copied.
2015-02-12 16:12:50,087 INFO reduce.MergeManagerImpl (MergeManagerImpl.java:finalMerge(673)) - finalMerge called with 2 in-memory map-outputs and 0 on-disk map-outputs
2015-02-12 16:12:50,089 INFO mapred.Merger (Merger.java:merge(606)) - Merging 2 sorted segments
2015-02-12 16:12:50,089 INFO mapred.Merger (Merger.java:merge(705)) - Down to the last merge-pass, with 2 segments left of total size: 220 bytes
2015-02-12 16:12:50,090 INFO reduce.MergeManagerImpl (MergeManagerImpl.java:finalMerge(750)) - Merged 2 segments, 238 bytes to disk to satisfy reduce memory limit
2015-02-12 16:12:50,091 INFO reduce.MergeManagerImpl (MergeManagerImpl.java:finalMerge(780)) - Merging 1 files, 240 bytes from disk
2015-02-12 16:12:50,091 INFO reduce.MergeManagerImpl (MergeManagerImpl.java:finalMerge(795)) - Merging 0 segments, 0 bytes from memory into reduce
2015-02-12 16:12:50,091 INFO mapred.Merger (Merger.java:merge(606)) - Merging 1 sorted segments
2015-02-12 16:12:50,091 INFO mapred.Merger (Merger.java:merge(705)) - Down to the last merge-pass, with 1 segments left of total size: 227 bytes
2015-02-12 16:12:50,092 INFO mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) - 2 / 2 copied.
2015-02-12 16:12:50,092 INFO hadoop.FlowReducer (FlowReducer.java:configure(78)) - cascading version: 2.6.3
2015-02-12 16:12:50,092 INFO hadoop.FlowReducer (FlowReducer.java:configure(79)) - child jvm opts: -Xmx200m
2015-02-12 16:12:50,097 INFO hadoop.FlowReducer (FlowReducer.java:configure(97)) - sourcing from: GroupBy(sum-lhs+sum-rhs)[by: sum-lhs:[{1}:'char'] sum-rhs:[{1}:'char']]
2015-02-12 16:12:50,097 INFO hadoop.FlowReducer (FlowReducer.java:configure(100)) - sinking to: Hfs["TextDelimited[['char', 'sum']]"]["/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/test/output/hadoop2-mr1/assemblyhelpersplatform/mergesum"]
2015-02-12 16:12:50,106 INFO mapred.Task (Task.java:done(1001)) - Task:attempt_local1316284914_0016_r_000000_0 is done. And is in the process of committing
2015-02-12 16:12:50,107 INFO mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) - 2 / 2 copied.
2015-02-12 16:12:50,107 INFO mapred.Task (Task.java:commit(1162)) - Task attempt_local1316284914_0016_r_000000_0 is allowed to commit now
2015-02-12 16:12:50,108 INFO output.FileOutputCommitter (FileOutputCommitter.java:commitTask(439)) - Saved output of task 'attempt_local1316284914_0016_r_000000_0' to file:/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/test/output/hadoop2-mr1/assemblyhelpersplatform/mergesum/_temporary/0/task_local1316284914_0016_r_000000
2015-02-12 16:12:50,109 INFO mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) - reduce > reduce
2015-02-12 16:12:50,109 INFO mapred.Task (Task.java:sendDone(1121)) - Task 'attempt_local1316284914_0016_r_000000_0' done.
2015-02-12 16:12:50,109 INFO mapred.LocalJobRunner (LocalJobRunner.java:run(325)) - Finishing task: attempt_local1316284914_0016_r_000000_0
2015-02-12 16:12:50,109 INFO mapred.LocalJobRunner (LocalJobRunner.java:runTasks(456)) - reduce task executor complete.
2015-02-12 16:12:50,124 INFO util.Hadoop18TapUtil (Hadoop18TapUtil.java:cleanTempPath(219)) - deleting temp path /Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/test/output/hadoop2-mr1/assemblyhelpersplatform/mergesum/_temporary
2015-02-12 16:12:50,148 INFO mapred.FileInputFormat (FileInputFormat.java:listStatus(247)) - Total input paths to process : 1
2015-02-12 16:12:50,171 INFO mapred.FileInputFormat (FileInputFormat.java:listStatus(247)) - Total input paths to process : 1
2015-02-12 16:12:50,172 INFO cascading.PlatformTestCase (PlatformTestCase.java:tearDown(209)) - copying to local /Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/test/output//hadoop2-mr1/assemblyhelpersplatform/mergesum
2015-02-12 16:12:50,191 INFO util.HadoopUtil (HadoopUtil.java:findMainClass(336)) - using default application jar, may cause class not found exceptions on the cluster
2015-02-12 16:12:50,192 INFO planner.HadoopPlanner (HadoopPlanner.java:initialize(225)) - using application jar: /Users/soberoi/.gradle/caches/modules-2/files-2.1/cascading/cascading-hadoop2-mr1/2.6.3/9ae016e2b8d246b9c1de9168e715d9f1b971aa4a/cascading-hadoop2-mr1-2.6.3.jar
2015-02-12 16:12:50,196 INFO Configuration.deprecation (Configuration.java:warnOnceIfDeprecated(1022)) - mapred.used.genericoptionsparser is deprecated. Instead, use mapreduce.client.genericoptionsparser.used
2015-02-12 16:12:50,197 INFO Configuration.deprecation (Configuration.java:warnOnceIfDeprecated(1022)) - mapred.job.tracker is deprecated. Instead, use mapreduce.jobtracker.address
2015-02-12 16:12:50,197 INFO Configuration.deprecation (Configuration.java:warnOnceIfDeprecated(1022)) - mapred.output.compress is deprecated. Instead, use mapreduce.output.fileoutputformat.compress
2015-02-12 16:12:50,202 INFO flow.Flow (BaseFlow.java:logInfo(1378)) - [shape] starting
2015-02-12 16:12:50,202 INFO flow.Flow (BaseFlow.java:logInfo(1378)) - [shape] source: Hfs["TextLine[['offset', 'line']->[ALL]]"]["/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/classes/test/data/lower.txt"]
2015-02-12 16:12:50,202 INFO flow.Flow (BaseFlow.java:logInfo(1378)) - [shape] sink: Hfs["TextLine[['num']]"]["/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/test/output/hadoop2-mr1/assemblyhelpersplatform/discardnarrow"]
2015-02-12 16:12:50,202 INFO flow.Flow (BaseFlow.java:logInfo(1378)) - [shape] parallel execution is enabled: false
2015-02-12 16:12:50,202 INFO flow.Flow (BaseFlow.java:logInfo(1378)) - [shape] starting jobs: 1
2015-02-12 16:12:50,202 INFO flow.Flow (BaseFlow.java:logInfo(1378)) - [shape] allocating threads: 1
2015-02-12 16:12:50,203 INFO flow.FlowStep (BaseFlowStep.java:logInfo(834)) - [shape] starting step: (1/1) ...ersplatform/discardnarrow
2015-02-12 16:12:50,203 INFO jvm.JvmMetrics (JvmMetrics.java:init(71)) - Cannot initialize JVM Metrics with processName=JobTracker, sessionId= - already initialized
2015-02-12 16:12:50,204 INFO jvm.JvmMetrics (JvmMetrics.java:init(71)) - Cannot initialize JVM Metrics with processName=JobTracker, sessionId= - already initialized
2015-02-12 16:12:50,222 INFO mapred.FileInputFormat (FileInputFormat.java:listStatus(247)) - Total input paths to process : 1
2015-02-12 16:12:50,238 INFO mapreduce.JobSubmitter (JobSubmitter.java:submitJobInternal(492)) - number of splits:1
2015-02-12 16:12:50,249 INFO mapreduce.JobSubmitter (JobSubmitter.java:printTokens(575)) - Submitting tokens for job: job_local1214593860_0017
2015-02-12 16:12:50,261 WARN conf.Configuration (Configuration.java:loadProperty(2510)) - file:/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/tmp/cascading/staging/soberoi1214593860/.staging/job_local1214593860_0017/job.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.retry.interval; Ignoring.
2015-02-12 16:12:50,262 WARN conf.Configuration (Configuration.java:loadProperty(2510)) - file:/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/tmp/cascading/staging/soberoi1214593860/.staging/job_local1214593860_0017/job.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.attempts; Ignoring.
2015-02-12 16:12:50,293 WARN conf.Configuration (Configuration.java:loadProperty(2510)) - file:/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/test/tmp/mapred/local/localRunner/soberoi/job_local1214593860_0017/job_local1214593860_0017.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.retry.interval; Ignoring.
2015-02-12 16:12:50,294 WARN conf.Configuration (Configuration.java:loadProperty(2510)) - file:/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/test/tmp/mapred/local/localRunner/soberoi/job_local1214593860_0017/job_local1214593860_0017.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.attempts; Ignoring.
2015-02-12 16:12:50,294 INFO mapreduce.Job (Job.java:submit(1299)) - The url to track the job: http://localhost:8080/
2015-02-12 16:12:50,294 INFO mapred.LocalJobRunner (LocalJobRunner.java:createOutputCommitter(471)) - OutputCommitter set in config null
2015-02-12 16:12:50,294 INFO flow.FlowStep (BaseFlowStep.java:logInfo(834)) - [shape] submitted hadoop job: job_local1214593860_0017
2015-02-12 16:12:50,295 INFO flow.FlowStep (BaseFlowStep.java:logInfo(834)) - [shape] tracking url: http://localhost:8080/
2015-02-12 16:12:50,295 INFO mapred.LocalJobRunner (LocalJobRunner.java:createOutputCommitter(489)) - OutputCommitter is org.apache.hadoop.mapred.FileOutputCommitter
2015-02-12 16:12:50,296 INFO mapred.LocalJobRunner (LocalJobRunner.java:runTasks(448)) - Waiting for map tasks
2015-02-12 16:12:50,296 INFO mapred.LocalJobRunner (LocalJobRunner.java:run(224)) - Starting task: attempt_local1214593860_0017_m_000000_0
2015-02-12 16:12:50,297 INFO util.ProcfsBasedProcessTree (ProcfsBasedProcessTree.java:isAvailable(181)) - ProcfsBasedProcessTree currently is supported only on Linux.
2015-02-12 16:12:50,297 INFO mapred.Task (Task.java:initialize(587)) - Using ResourceCalculatorProcessTree : null
2015-02-12 16:12:50,298 INFO io.MultiInputSplit (MultiInputSplit.java:readFields(161)) - current split input path: file:/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/classes/test/data/lower.txt
2015-02-12 16:12:50,298 INFO mapred.MapTask (MapTask.java:updateJobWithSplit(482)) - Processing split: cascading.tap.hadoop.io.MultiInputSplit@221cf05
2015-02-12 16:12:50,299 INFO mapred.MapTask (MapTask.java:runOldMapper(436)) - numReduceTasks: 0
2015-02-12 16:12:50,304 INFO hadoop.FlowMapper (FlowMapper.java:configure(71)) - cascading version: 2.6.3
2015-02-12 16:12:50,304 INFO hadoop.FlowMapper (FlowMapper.java:configure(72)) - child jvm opts: -Xmx200m
2015-02-12 16:12:50,307 INFO hadoop.FlowMapper (FlowMapper.java:configure(87)) - sourcing from: Hfs["TextLine[['offset', 'line']->[ALL]]"]["/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/classes/test/data/lower.txt"]
2015-02-12 16:12:50,307 INFO hadoop.FlowMapper (FlowMapper.java:configure(90)) - sinking to: Hfs["TextLine[['num']]"]["/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/test/output/hadoop2-mr1/assemblyhelpersplatform/discardnarrow"]
2015-02-12 16:12:50,308 INFO mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) -
2015-02-12 16:12:50,309 INFO mapred.Task (Task.java:done(1001)) - Task:attempt_local1214593860_0017_m_000000_0 is done. And is in the process of committing
2015-02-12 16:12:50,310 INFO mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) -
2015-02-12 16:12:50,310 INFO mapred.Task (Task.java:commit(1162)) - Task attempt_local1214593860_0017_m_000000_0 is allowed to commit now
2015-02-12 16:12:50,310 INFO output.FileOutputCommitter (FileOutputCommitter.java:commitTask(439)) - Saved output of task 'attempt_local1214593860_0017_m_000000_0' to file:/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/test/output/hadoop2-mr1/assemblyhelpersplatform/discardnarrow/_temporary/0/task_local1214593860_0017_m_000000
2015-02-12 16:12:50,311 INFO mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) - file:/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/classes/test/data/lower.txt:0+19
2015-02-12 16:12:50,311 INFO mapred.Task (Task.java:sendDone(1121)) - Task 'attempt_local1214593860_0017_m_000000_0' done.
2015-02-12 16:12:50,311 INFO mapred.LocalJobRunner (LocalJobRunner.java:run(249)) - Finishing task: attempt_local1214593860_0017_m_000000_0
2015-02-12 16:12:50,311 INFO mapred.LocalJobRunner (LocalJobRunner.java:runTasks(456)) - map task executor complete.
2015-02-12 16:12:50,322 INFO util.Hadoop18TapUtil (Hadoop18TapUtil.java:cleanTempPath(219)) - deleting temp path /Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/test/output/hadoop2-mr1/assemblyhelpersplatform/discardnarrow/_temporary
2015-02-12 16:12:50,350 INFO mapred.FileInputFormat (FileInputFormat.java:listStatus(247)) - Total input paths to process : 1
2015-02-12 16:12:50,351 INFO cascading.PlatformTestCase (PlatformTestCase.java:tearDown(209)) - copying to local /Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/test/output//hadoop2-mr1/assemblyhelpersplatform/discardnarrow
2015-02-12 16:12:50,362 INFO util.HadoopUtil (HadoopUtil.java:findMainClass(336)) - using default application jar, may cause class not found exceptions on the cluster
2015-02-12 16:12:50,362 INFO planner.HadoopPlanner (HadoopPlanner.java:initialize(225)) - using application jar: /Users/soberoi/.gradle/caches/modules-2/files-2.1/cascading/cascading-hadoop2-mr1/2.6.3/9ae016e2b8d246b9c1de9168e715d9f1b971aa4a/cascading-hadoop2-mr1-2.6.3.jar
2015-02-12 16:12:50,367 INFO Configuration.deprecation (Configuration.java:warnOnceIfDeprecated(1022)) - mapred.used.genericoptionsparser is deprecated. Instead, use mapreduce.client.genericoptionsparser.used
2015-02-12 16:12:50,369 INFO Configuration.deprecation (Configuration.java:warnOnceIfDeprecated(1022)) - mapred.job.tracker is deprecated. Instead, use mapreduce.jobtracker.address
2015-02-12 16:12:50,369 INFO Configuration.deprecation (Configuration.java:warnOnceIfDeprecated(1022)) - mapred.output.compress is deprecated. Instead, use mapreduce.output.fileoutputformat.compress
2015-02-12 16:12:50,374 INFO flow.Flow (BaseFlow.java:logInfo(1378)) - [shape] starting
2015-02-12 16:12:50,375 INFO flow.Flow (BaseFlow.java:logInfo(1378)) - [shape] source: Hfs["TextLine[['offset', 'line']->[ALL]]"]["/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/classes/test/data/lower.txt"]
2015-02-12 16:12:50,375 INFO flow.Flow (BaseFlow.java:logInfo(1378)) - [shape] sink: Hfs["TextLine[['line']->['item', 'element']]"]["/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/test/output/hadoop2-mr1/assemblyhelpersplatform/rename"]
2015-02-12 16:12:50,375 INFO flow.Flow (BaseFlow.java:logInfo(1378)) - [shape] parallel execution is enabled: false
2015-02-12 16:12:50,375 INFO flow.Flow (BaseFlow.java:logInfo(1378)) - [shape] starting jobs: 1
2015-02-12 16:12:50,375 INFO flow.Flow (BaseFlow.java:logInfo(1378)) - [shape] allocating threads: 1
2015-02-12 16:12:50,376 INFO flow.FlowStep (BaseFlowStep.java:logInfo(834)) - [shape] starting step: (1/1) ...blyhelpersplatform/rename
2015-02-12 16:12:50,377 INFO jvm.JvmMetrics (JvmMetrics.java:init(71)) - Cannot initialize JVM Metrics with processName=JobTracker, sessionId= - already initialized
2015-02-12 16:12:50,378 INFO jvm.JvmMetrics (JvmMetrics.java:init(71)) - Cannot initialize JVM Metrics with processName=JobTracker, sessionId= - already initialized
2015-02-12 16:12:50,400 INFO mapred.FileInputFormat (FileInputFormat.java:listStatus(247)) - Total input paths to process : 1
2015-02-12 16:12:50,416 INFO mapreduce.JobSubmitter (JobSubmitter.java:submitJobInternal(492)) - number of splits:1
2015-02-12 16:12:50,427 INFO mapreduce.JobSubmitter (JobSubmitter.java:printTokens(575)) - Submitting tokens for job: job_local90064939_0018
2015-02-12 16:12:50,441 WARN conf.Configuration (Configuration.java:loadProperty(2510)) - file:/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/tmp/cascading/staging/soberoi90064939/.staging/job_local90064939_0018/job.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.retry.interval; Ignoring.
2015-02-12 16:12:50,441 WARN conf.Configuration (Configuration.java:loadProperty(2510)) - file:/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/tmp/cascading/staging/soberoi90064939/.staging/job_local90064939_0018/job.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.attempts; Ignoring.
2015-02-12 16:12:50,475 WARN conf.Configuration (Configuration.java:loadProperty(2510)) - file:/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/test/tmp/mapred/local/localRunner/soberoi/job_local90064939_0018/job_local90064939_0018.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.retry.interval; Ignoring.
2015-02-12 16:12:50,477 WARN conf.Configuration (Configuration.java:loadProperty(2510)) - file:/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/test/tmp/mapred/local/localRunner/soberoi/job_local90064939_0018/job_local90064939_0018.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.attempts; Ignoring.
2015-02-12 16:12:50,477 INFO mapreduce.Job (Job.java:submit(1299)) - The url to track the job: http://localhost:8080/
2015-02-12 16:12:50,478 INFO mapred.LocalJobRunner (LocalJobRunner.java:createOutputCommitter(471)) - OutputCommitter set in config null
2015-02-12 16:12:50,478 INFO mapred.LocalJobRunner (LocalJobRunner.java:createOutputCommitter(489)) - OutputCommitter is org.apache.hadoop.mapred.FileOutputCommitter
2015-02-12 16:12:50,478 INFO flow.FlowStep (BaseFlowStep.java:logInfo(834)) - [shape] submitted hadoop job: job_local90064939_0018
2015-02-12 16:12:50,478 INFO flow.FlowStep (BaseFlowStep.java:logInfo(834)) - [shape] tracking url: http://localhost:8080/
2015-02-12 16:12:50,480 INFO mapred.LocalJobRunner (LocalJobRunner.java:runTasks(448)) - Waiting for map tasks
2015-02-12 16:12:50,480 INFO mapred.LocalJobRunner (LocalJobRunner.java:run(224)) - Starting task: attempt_local90064939_0018_m_000000_0
2015-02-12 16:12:50,481 INFO util.ProcfsBasedProcessTree (ProcfsBasedProcessTree.java:isAvailable(181)) - ProcfsBasedProcessTree currently is supported only on Linux.
2015-02-12 16:12:50,481 INFO mapred.Task (Task.java:initialize(587)) - Using ResourceCalculatorProcessTree : null
2015-02-12 16:12:50,482 INFO io.MultiInputSplit (MultiInputSplit.java:readFields(161)) - current split input path: file:/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/classes/test/data/lower.txt
2015-02-12 16:12:50,482 INFO mapred.MapTask (MapTask.java:updateJobWithSplit(482)) - Processing split: cascading.tap.hadoop.io.MultiInputSplit@4dbe4157
2015-02-12 16:12:50,483 INFO mapred.MapTask (MapTask.java:runOldMapper(436)) - numReduceTasks: 0
2015-02-12 16:12:50,488 INFO hadoop.FlowMapper (FlowMapper.java:configure(71)) - cascading version: 2.6.3
2015-02-12 16:12:50,488 INFO hadoop.FlowMapper (FlowMapper.java:configure(72)) - child jvm opts: -Xmx200m
2015-02-12 16:12:50,491 INFO hadoop.FlowMapper (FlowMapper.java:configure(87)) - sourcing from: Hfs["TextLine[['offset', 'line']->[ALL]]"]["/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/classes/test/data/lower.txt"]
2015-02-12 16:12:50,491 INFO hadoop.FlowMapper (FlowMapper.java:configure(90)) - sinking to: Hfs["TextLine[['line']->['item', 'element']]"]["/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/test/output/hadoop2-mr1/assemblyhelpersplatform/rename"]
2015-02-12 16:12:50,492 INFO mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) -
2015-02-12 16:12:50,493 INFO mapred.Task (Task.java:done(1001)) - Task:attempt_local90064939_0018_m_000000_0 is done. And is in the process of committing
2015-02-12 16:12:50,494 INFO mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) -
2015-02-12 16:12:50,494 INFO mapred.Task (Task.java:commit(1162)) - Task attempt_local90064939_0018_m_000000_0 is allowed to commit now
2015-02-12 16:12:50,494 INFO output.FileOutputCommitter (FileOutputCommitter.java:commitTask(439)) - Saved output of task 'attempt_local90064939_0018_m_000000_0' to file:/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/test/output/hadoop2-mr1/assemblyhelpersplatform/rename/_temporary/0/task_local90064939_0018_m_000000
2015-02-12 16:12:50,495 INFO mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) - file:/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/classes/test/data/lower.txt:0+19
2015-02-12 16:12:50,495 INFO mapred.Task (Task.java:sendDone(1121)) - Task 'attempt_local90064939_0018_m_000000_0' done.
2015-02-12 16:12:50,496 INFO mapred.LocalJobRunner (LocalJobRunner.java:run(249)) - Finishing task: attempt_local90064939_0018_m_000000_0
2015-02-12 16:12:50,496 INFO mapred.LocalJobRunner (LocalJobRunner.java:runTasks(456)) - map task executor complete.
2015-02-12 16:12:50,504 INFO util.Hadoop18TapUtil (Hadoop18TapUtil.java:cleanTempPath(219)) - deleting temp path /Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/test/output/hadoop2-mr1/assemblyhelpersplatform/rename/_temporary
2015-02-12 16:12:50,524 INFO mapred.FileInputFormat (FileInputFormat.java:listStatus(247)) - Total input paths to process : 1
2015-02-12 16:12:50,525 INFO cascading.PlatformTestCase (PlatformTestCase.java:tearDown(209)) - copying to local /Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/test/output//hadoop2-mr1/assemblyhelpersplatform/rename
2015-02-12 16:12:50,538 INFO util.HadoopUtil (HadoopUtil.java:findMainClass(336)) - using default application jar, may cause class not found exceptions on the cluster
2015-02-12 16:12:50,539 INFO planner.HadoopPlanner (HadoopPlanner.java:initialize(225)) - using application jar: /Users/soberoi/.gradle/caches/modules-2/files-2.1/cascading/cascading-hadoop2-mr1/2.6.3/9ae016e2b8d246b9c1de9168e715d9f1b971aa4a/cascading-hadoop2-mr1-2.6.3.jar
2015-02-12 16:12:50,547 INFO Configuration.deprecation (Configuration.java:warnOnceIfDeprecated(1022)) - mapred.used.genericoptionsparser is deprecated. Instead, use mapreduce.client.genericoptionsparser.used
2015-02-12 16:12:50,548 INFO Configuration.deprecation (Configuration.java:warnOnceIfDeprecated(1022)) - mapred.job.tracker is deprecated. Instead, use mapreduce.jobtracker.address
2015-02-12 16:12:50,549 INFO Configuration.deprecation (Configuration.java:warnOnceIfDeprecated(1022)) - mapred.output.compress is deprecated. Instead, use mapreduce.output.fileoutputformat.compress
2015-02-12 16:12:50,556 INFO flow.Flow (BaseFlow.java:logInfo(1378)) - [name] starting
2015-02-12 16:12:50,557 INFO flow.Flow (BaseFlow.java:logInfo(1378)) - [name] source: Hfs["TextDelimited[['num', 'char']]"]["/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/classes/test/data/rhs.txt"]
2015-02-12 16:12:50,557 INFO flow.Flow (BaseFlow.java:logInfo(1378)) - [name] source: Hfs["TextDelimited[['num', 'char']]"]["/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/classes/test/data/lhs.txt"]
2015-02-12 16:12:50,557 INFO flow.Flow (BaseFlow.java:logInfo(1378)) - [name] sink: Hfs["TextDelimited[['char', 'sum', 'count', 'average']]"]["/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/test/output/hadoop2-mr1/assemblyhelpersplatform/multimerge+true"]
2015-02-12 16:12:50,557 INFO flow.Flow (BaseFlow.java:logInfo(1378)) - [name] parallel execution is enabled: false
2015-02-12 16:12:50,557 INFO flow.Flow (BaseFlow.java:logInfo(1378)) - [name] starting jobs: 1
2015-02-12 16:12:50,557 INFO flow.Flow (BaseFlow.java:logInfo(1378)) - [name] allocating threads: 1
2015-02-12 16:12:50,557 INFO flow.FlowStep (BaseFlowStep.java:logInfo(834)) - [name] starting step: (1/1) ...splatform/multimerge+true
2015-02-12 16:12:50,558 INFO jvm.JvmMetrics (JvmMetrics.java:init(71)) - Cannot initialize JVM Metrics with processName=JobTracker, sessionId= - already initialized
2015-02-12 16:12:50,559 INFO jvm.JvmMetrics (JvmMetrics.java:init(71)) - Cannot initialize JVM Metrics with processName=JobTracker, sessionId= - already initialized
2015-02-12 16:12:50,578 INFO mapred.FileInputFormat (FileInputFormat.java:listStatus(247)) - Total input paths to process : 1
2015-02-12 16:12:50,579 INFO mapred.FileInputFormat (FileInputFormat.java:listStatus(247)) - Total input paths to process : 1
2015-02-12 16:12:50,594 INFO mapreduce.JobSubmitter (JobSubmitter.java:submitJobInternal(492)) - number of splits:2
2015-02-12 16:12:50,605 INFO mapreduce.JobSubmitter (JobSubmitter.java:printTokens(575)) - Submitting tokens for job: job_local366847577_0019
2015-02-12 16:12:50,615 WARN conf.Configuration (Configuration.java:loadProperty(2510)) - file:/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/tmp/cascading/staging/soberoi366847577/.staging/job_local366847577_0019/job.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.retry.interval; Ignoring.
2015-02-12 16:12:50,616 WARN conf.Configuration (Configuration.java:loadProperty(2510)) - file:/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/tmp/cascading/staging/soberoi366847577/.staging/job_local366847577_0019/job.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.attempts; Ignoring.
2015-02-12 16:12:50,644 WARN conf.Configuration (Configuration.java:loadProperty(2510)) - file:/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/test/tmp/mapred/local/localRunner/soberoi/job_local366847577_0019/job_local366847577_0019.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.retry.interval; Ignoring.
2015-02-12 16:12:50,645 WARN conf.Configuration (Configuration.java:loadProperty(2510)) - file:/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/test/tmp/mapred/local/localRunner/soberoi/job_local366847577_0019/job_local366847577_0019.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.attempts; Ignoring.
2015-02-12 16:12:50,645 INFO mapreduce.Job (Job.java:submit(1299)) - The url to track the job: http://localhost:8080/
2015-02-12 16:12:50,646 INFO mapred.LocalJobRunner (LocalJobRunner.java:createOutputCommitter(471)) - OutputCommitter set in config null
2015-02-12 16:12:50,646 INFO mapred.LocalJobRunner (LocalJobRunner.java:createOutputCommitter(489)) - OutputCommitter is org.apache.hadoop.mapred.FileOutputCommitter
2015-02-12 16:12:50,646 INFO flow.FlowStep (BaseFlowStep.java:logInfo(834)) - [name] submitted hadoop job: job_local366847577_0019
2015-02-12 16:12:50,646 INFO flow.FlowStep (BaseFlowStep.java:logInfo(834)) - [name] tracking url: http://localhost:8080/
2015-02-12 16:12:50,648 INFO mapred.LocalJobRunner (LocalJobRunner.java:runTasks(448)) - Waiting for map tasks
2015-02-12 16:12:50,648 INFO mapred.LocalJobRunner (LocalJobRunner.java:run(224)) - Starting task: attempt_local366847577_0019_m_000000_0
2015-02-12 16:12:50,649 INFO util.ProcfsBasedProcessTree (ProcfsBasedProcessTree.java:isAvailable(181)) - ProcfsBasedProcessTree currently is supported only on Linux.
2015-02-12 16:12:50,649 INFO mapred.Task (Task.java:initialize(587)) - Using ResourceCalculatorProcessTree : null
2015-02-12 16:12:50,650 INFO io.MultiInputSplit (MultiInputSplit.java:readFields(161)) - current split input path: file:/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/classes/test/data/lhs.txt
2015-02-12 16:12:50,650 INFO mapred.MapTask (MapTask.java:updateJobWithSplit(482)) - Processing split: cascading.tap.hadoop.io.MultiInputSplit@3500b5d7
2015-02-12 16:12:50,651 INFO mapred.MapTask (MapTask.java:runOldMapper(436)) - numReduceTasks: 1
2015-02-12 16:12:50,667 INFO mapred.MapTask (MapTask.java:setEquator(1202)) - (EQUATOR) 0 kvi 26214396(104857584)
2015-02-12 16:12:50,668 INFO mapred.MapTask (MapTask.java:init(995)) - mapreduce.task.io.sort.mb: 100
2015-02-12 16:12:50,668 INFO mapred.MapTask (MapTask.java:init(996)) - soft limit at 83886080
2015-02-12 16:12:50,668 INFO mapred.MapTask (MapTask.java:init(997)) - bufstart = 0; bufvoid = 104857600
2015-02-12 16:12:50,668 INFO mapred.MapTask (MapTask.java:init(998)) - kvstart = 26214396; length = 6553600
2015-02-12 16:12:50,669 INFO mapred.MapTask (MapTask.java:createSortingCollector(402)) - Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer
2015-02-12 16:12:50,669 INFO hadoop.FlowMapper (FlowMapper.java:configure(71)) - cascading version: 2.6.3
2015-02-12 16:12:50,669 INFO hadoop.FlowMapper (FlowMapper.java:configure(72)) - child jvm opts: -Xmx200m
2015-02-12 16:12:50,676 INFO hadoop.FlowMapper (FlowMapper.java:configure(87)) - sourcing from: Hfs["TextDelimited[['num', 'char']]"]["/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/classes/test/data/lhs.txt"]
2015-02-12 16:12:50,676 INFO hadoop.FlowMapper (FlowMapper.java:configure(90)) - sinking to: GroupBy(name)[by:[{1}:'char']]
2015-02-12 16:12:50,676 INFO assembly.AggregateBy (AggregateBy.java:prepare(267)) - using threshold value: 2
2015-02-12 16:12:50,677 INFO assembly.AggregateBy (AggregateBy.java:removeEldestEntry(327)) - flushed keys num times: 1, with threshold: 2
2015-02-12 16:12:50,677 INFO assembly.AggregateBy (AggregateBy.java:removeEldestEntry(328)) - mem on flush (mb), free: 195, total: 509, max: 672
2015-02-12 16:12:50,677 INFO assembly.AggregateBy (AggregateBy.java:removeEldestEntry(333)) - total mem is 75% of max mem, to better utilize unused memory consider increasing current LRU threshold with system property "cascading.aggregateby.threshold"
2015-02-12 16:12:50,678 INFO assembly.AggregateBy (AggregateBy.java:removeEldestEntry(327)) - flushed keys num times: 3, with threshold: 2
2015-02-12 16:12:50,678 INFO assembly.AggregateBy (AggregateBy.java:removeEldestEntry(328)) - mem on flush (mb), free: 195, total: 509, max: 672
2015-02-12 16:12:50,678 INFO assembly.AggregateBy (AggregateBy.java:removeEldestEntry(333)) - total mem is 75% of max mem, to better utilize unused memory consider increasing current LRU threshold with system property "cascading.aggregateby.threshold"
2015-02-12 16:12:50,679 INFO assembly.AggregateBy (AggregateBy.java:removeEldestEntry(327)) - flushed keys num times: 5, with threshold: 2
2015-02-12 16:12:50,679 INFO assembly.AggregateBy (AggregateBy.java:removeEldestEntry(328)) - mem on flush (mb), free: 195, total: 509, max: 672
2015-02-12 16:12:50,679 INFO assembly.AggregateBy (AggregateBy.java:removeEldestEntry(333)) - total mem is 75% of max mem, to better utilize unused memory consider increasing current LRU threshold with system property "cascading.aggregateby.threshold"
2015-02-12 16:12:50,679 INFO assembly.AggregateBy (AggregateBy.java:removeEldestEntry(327)) - flushed keys num times: 7, with threshold: 2
2015-02-12 16:12:50,679 INFO assembly.AggregateBy (AggregateBy.java:removeEldestEntry(328)) - mem on flush (mb), free: 195, total: 509, max: 672
2015-02-12 16:12:50,680 INFO assembly.AggregateBy (AggregateBy.java:removeEldestEntry(333)) - total mem is 75% of max mem, to better utilize unused memory consider increasing current LRU threshold with system property "cascading.aggregateby.threshold"
2015-02-12 16:12:50,681 INFO mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) -
2015-02-12 16:12:50,681 INFO mapred.MapTask (MapTask.java:flush(1457)) - Starting flush of map output
2015-02-12 16:12:50,681 INFO mapred.MapTask (MapTask.java:flush(1475)) - Spilling map output
2015-02-12 16:12:50,681 INFO mapred.MapTask (MapTask.java:flush(1476)) - bufstart = 0; bufend = 216; bufvoid = 104857600
2015-02-12 16:12:50,681 INFO mapred.MapTask (MapTask.java:flush(1478)) - kvstart = 26214396(104857584); kvend = 26214364(104857456); length = 33/6553600
2015-02-12 16:12:50,684 INFO mapred.MapTask (MapTask.java:sortAndSpill(1660)) - Finished spill 0
2015-02-12 16:12:50,685 INFO mapred.Task (Task.java:done(1001)) - Task:attempt_local366847577_0019_m_000000_0 is done. And is in the process of committing
2015-02-12 16:12:50,687 INFO mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) - file:/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/classes/test/data/lhs.txt:0+52
2015-02-12 16:12:50,687 INFO mapred.Task (Task.java:sendDone(1121)) - Task 'attempt_local366847577_0019_m_000000_0' done.
2015-02-12 16:12:50,687 INFO mapred.LocalJobRunner (LocalJobRunner.java:run(249)) - Finishing task: attempt_local366847577_0019_m_000000_0
2015-02-12 16:12:50,687 INFO mapred.LocalJobRunner (LocalJobRunner.java:run(224)) - Starting task: attempt_local366847577_0019_m_000001_0
2015-02-12 16:12:50,688 INFO util.ProcfsBasedProcessTree (ProcfsBasedProcessTree.java:isAvailable(181)) - ProcfsBasedProcessTree currently is supported only on Linux.
2015-02-12 16:12:50,688 INFO mapred.Task (Task.java:initialize(587)) - Using ResourceCalculatorProcessTree : null
2015-02-12 16:12:50,689 INFO io.MultiInputSplit (MultiInputSplit.java:readFields(161)) - current split input path: file:/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/classes/test/data/rhs.txt
2015-02-12 16:12:50,689 INFO mapred.MapTask (MapTask.java:updateJobWithSplit(482)) - Processing split: cascading.tap.hadoop.io.MultiInputSplit@59f4e49e
2015-02-12 16:12:50,690 INFO mapred.MapTask (MapTask.java:runOldMapper(436)) - numReduceTasks: 1
2015-02-12 16:12:50,701 INFO mapred.MapTask (MapTask.java:setEquator(1202)) - (EQUATOR) 0 kvi 26214396(104857584)
2015-02-12 16:12:50,701 INFO mapred.MapTask (MapTask.java:init(995)) - mapreduce.task.io.sort.mb: 100
2015-02-12 16:12:50,702 INFO mapred.MapTask (MapTask.java:init(996)) - soft limit at 83886080
2015-02-12 16:12:50,702 INFO mapred.MapTask (MapTask.java:init(997)) - bufstart = 0; bufvoid = 104857600
2015-02-12 16:12:50,702 INFO mapred.MapTask (MapTask.java:init(998)) - kvstart = 26214396; length = 6553600
2015-02-12 16:12:50,703 INFO mapred.MapTask (MapTask.java:createSortingCollector(402)) - Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer
2015-02-12 16:12:50,703 INFO hadoop.FlowMapper (FlowMapper.java:configure(71)) - cascading version: 2.6.3
2015-02-12 16:12:50,703 INFO hadoop.FlowMapper (FlowMapper.java:configure(72)) - child jvm opts: -Xmx200m
2015-02-12 16:12:50,709 INFO hadoop.FlowMapper (FlowMapper.java:configure(87)) - sourcing from: Hfs["TextDelimited[['num', 'char']]"]["/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/classes/test/data/rhs.txt"]
2015-02-12 16:12:50,709 INFO hadoop.FlowMapper (FlowMapper.java:configure(90)) - sinking to: GroupBy(name)[by:[{1}:'char']]
2015-02-12 16:12:50,709 INFO assembly.AggregateBy (AggregateBy.java:prepare(267)) - using threshold value: 2
2015-02-12 16:12:50,712 INFO assembly.AggregateBy (AggregateBy.java:removeEldestEntry(327)) - flushed keys num times: 1, with threshold: 2
2015-02-12 16:12:50,712 INFO assembly.AggregateBy (AggregateBy.java:removeEldestEntry(328)) - mem on flush (mb), free: 93, total: 509, max: 672
2015-02-12 16:12:50,712 INFO assembly.AggregateBy (AggregateBy.java:removeEldestEntry(333)) - total mem is 75% of max mem, to better utilize unused memory consider increasing current LRU threshold with system property "cascading.aggregateby.threshold"
2015-02-12 16:12:50,713 INFO assembly.AggregateBy (AggregateBy.java:removeEldestEntry(327)) - flushed keys num times: 3, with threshold: 2
2015-02-12 16:12:50,713 INFO assembly.AggregateBy (AggregateBy.java:removeEldestEntry(328)) - mem on flush (mb), free: 93, total: 509, max: 672
2015-02-12 16:12:50,713 INFO assembly.AggregateBy (AggregateBy.java:removeEldestEntry(333)) - total mem is 75% of max mem, to better utilize unused memory consider increasing current LRU threshold with system property "cascading.aggregateby.threshold"
2015-02-12 16:12:50,713 INFO assembly.AggregateBy (AggregateBy.java:removeEldestEntry(327)) - flushed keys num times: 5, with threshold: 2
2015-02-12 16:12:50,713 INFO assembly.AggregateBy (AggregateBy.java:removeEldestEntry(328)) - mem on flush (mb), free: 93, total: 509, max: 672
2015-02-12 16:12:50,713 INFO assembly.AggregateBy (AggregateBy.java:removeEldestEntry(333)) - total mem is 75% of max mem, to better utilize unused memory consider increasing current LRU threshold with system property "cascading.aggregateby.threshold"
2015-02-12 16:12:50,714 INFO assembly.AggregateBy (AggregateBy.java:removeEldestEntry(327)) - flushed keys num times: 7, with threshold: 2
2015-02-12 16:12:50,714 INFO assembly.AggregateBy (AggregateBy.java:removeEldestEntry(328)) - mem on flush (mb), free: 93, total: 509, max: 672
2015-02-12 16:12:50,714 INFO assembly.AggregateBy (AggregateBy.java:removeEldestEntry(333)) - total mem is 75% of max mem, to better utilize unused memory consider increasing current LRU threshold with system property "cascading.aggregateby.threshold"
2015-02-12 16:12:50,715 INFO mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) -
2015-02-12 16:12:50,715 INFO mapred.MapTask (MapTask.java:flush(1457)) - Starting flush of map output
2015-02-12 16:12:50,715 INFO mapred.MapTask (MapTask.java:flush(1475)) - Spilling map output
2015-02-12 16:12:50,715 INFO mapred.MapTask (MapTask.java:flush(1476)) - bufstart = 0; bufend = 216; bufvoid = 104857600
2015-02-12 16:12:50,715 INFO mapred.MapTask (MapTask.java:flush(1478)) - kvstart = 26214396(104857584); kvend = 26214364(104857456); length = 33/6553600
2015-02-12 16:12:50,717 INFO mapred.MapTask (MapTask.java:sortAndSpill(1660)) - Finished spill 0
2015-02-12 16:12:50,718 INFO mapred.Task (Task.java:done(1001)) - Task:attempt_local366847577_0019_m_000001_0 is done. And is in the process of committing
2015-02-12 16:12:50,719 INFO mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) - file:/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/classes/test/data/rhs.txt:0+51
2015-02-12 16:12:50,720 INFO mapred.Task (Task.java:sendDone(1121)) - Task 'attempt_local366847577_0019_m_000001_0' done.
2015-02-12 16:12:50,720 INFO mapred.LocalJobRunner (LocalJobRunner.java:run(249)) - Finishing task: attempt_local366847577_0019_m_000001_0
2015-02-12 16:12:50,720 INFO mapred.LocalJobRunner (LocalJobRunner.java:runTasks(456)) - map task executor complete.
2015-02-12 16:12:50,720 INFO mapred.LocalJobRunner (LocalJobRunner.java:runTasks(448)) - Waiting for reduce tasks
2015-02-12 16:12:50,720 INFO mapred.LocalJobRunner (LocalJobRunner.java:run(302)) - Starting task: attempt_local366847577_0019_r_000000_0
2015-02-12 16:12:50,721 INFO util.ProcfsBasedProcessTree (ProcfsBasedProcessTree.java:isAvailable(181)) - ProcfsBasedProcessTree currently is supported only on Linux.
2015-02-12 16:12:50,721 INFO mapred.Task (Task.java:initialize(587)) - Using ResourceCalculatorProcessTree : null
2015-02-12 16:12:50,721 INFO mapred.ReduceTask (ReduceTask.java:run(362)) - Using ShuffleConsumerPlugin: org.apache.hadoop.mapreduce.task.reduce.Shuffle@497acafa
2015-02-12 16:12:50,722 INFO reduce.MergeManagerImpl (MergeManagerImpl.java:<init>(195)) - MergerManager: memoryLimit=493250144, maxSingleShuffleLimit=123312536, mergeThreshold=325545120, ioSortFactor=10, memToMemMergeOutputsThreshold=10
2015-02-12 16:12:50,722 INFO reduce.EventFetcher (EventFetcher.java:run(61)) - attempt_local366847577_0019_r_000000_0 Thread started: EventFetcher for fetching Map Completion Events
2015-02-12 16:12:50,723 INFO reduce.LocalFetcher (LocalFetcher.java:copyMapOutput(141)) - localfetcher#15 about to shuffle output of map attempt_local366847577_0019_m_000000_0 decomp: 236 len: 240 to MEMORY
2015-02-12 16:12:50,724 INFO reduce.InMemoryMapOutput (InMemoryMapOutput.java:shuffle(100)) - Read 236 bytes from map-output for attempt_local366847577_0019_m_000000_0
2015-02-12 16:12:50,724 INFO reduce.MergeManagerImpl (MergeManagerImpl.java:closeInMemoryFile(313)) - closeInMemoryFile -> map-output of size: 236, inMemoryMapOutputs.size() -> 1, commitMemory -> 0, usedMemory ->236
2015-02-12 16:12:50,725 INFO reduce.LocalFetcher (LocalFetcher.java:copyMapOutput(141)) - localfetcher#15 about to shuffle output of map attempt_local366847577_0019_m_000001_0 decomp: 236 len: 240 to MEMORY
2015-02-12 16:12:50,725 INFO reduce.InMemoryMapOutput (InMemoryMapOutput.java:shuffle(100)) - Read 236 bytes from map-output for attempt_local366847577_0019_m_000001_0
2015-02-12 16:12:50,725 INFO reduce.MergeManagerImpl (MergeManagerImpl.java:closeInMemoryFile(313)) - closeInMemoryFile -> map-output of size: 236, inMemoryMapOutputs.size() -> 2, commitMemory -> 236, usedMemory ->472
2015-02-12 16:12:50,725 INFO reduce.EventFetcher (EventFetcher.java:run(76)) - EventFetcher is interrupted.. Returning
2015-02-12 16:12:50,726 INFO mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) - 2 / 2 copied.
2015-02-12 16:12:50,726 INFO reduce.MergeManagerImpl (MergeManagerImpl.java:finalMerge(673)) - finalMerge called with 2 in-memory map-outputs and 0 on-disk map-outputs
2015-02-12 16:12:50,728 INFO mapred.Merger (Merger.java:merge(606)) - Merging 2 sorted segments
2015-02-12 16:12:50,728 INFO mapred.Merger (Merger.java:merge(705)) - Down to the last merge-pass, with 2 segments left of total size: 454 bytes
2015-02-12 16:12:50,730 INFO reduce.MergeManagerImpl (MergeManagerImpl.java:finalMerge(750)) - Merged 2 segments, 472 bytes to disk to satisfy reduce memory limit
2015-02-12 16:12:50,730 INFO reduce.MergeManagerImpl (MergeManagerImpl.java:finalMerge(780)) - Merging 1 files, 474 bytes from disk
2015-02-12 16:12:50,730 INFO reduce.MergeManagerImpl (MergeManagerImpl.java:finalMerge(795)) - Merging 0 segments, 0 bytes from memory into reduce
2015-02-12 16:12:50,730 INFO mapred.Merger (Merger.java:merge(606)) - Merging 1 sorted segments
2015-02-12 16:12:50,731 INFO mapred.Merger (Merger.java:merge(705)) - Down to the last merge-pass, with 1 segments left of total size: 461 bytes
2015-02-12 16:12:50,731 INFO mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) - 2 / 2 copied.
2015-02-12 16:12:50,732 INFO hadoop.FlowReducer (FlowReducer.java:configure(78)) - cascading version: 2.6.3
2015-02-12 16:12:50,732 INFO hadoop.FlowReducer (FlowReducer.java:configure(79)) - child jvm opts: -Xmx200m
2015-02-12 16:12:50,737 INFO hadoop.FlowReducer (FlowReducer.java:configure(97)) - sourcing from: GroupBy(name)[by:[{1}:'char']]
2015-02-12 16:12:50,737 INFO hadoop.FlowReducer (FlowReducer.java:configure(100)) - sinking to: Hfs["TextDelimited[['char', 'sum', 'count', 'average']]"]["/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/test/output/hadoop2-mr1/assemblyhelpersplatform/multimerge+true"]
2015-02-12 16:12:50,746 INFO mapred.Task (Task.java:done(1001)) - Task:attempt_local366847577_0019_r_000000_0 is done. And is in the process of committing
2015-02-12 16:12:50,747 INFO mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) - 2 / 2 copied.
2015-02-12 16:12:50,747 INFO mapred.Task (Task.java:commit(1162)) - Task attempt_local366847577_0019_r_000000_0 is allowed to commit now
2015-02-12 16:12:50,748 INFO output.FileOutputCommitter (FileOutputCommitter.java:commitTask(439)) - Saved output of task 'attempt_local366847577_0019_r_000000_0' to file:/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/test/output/hadoop2-mr1/assemblyhelpersplatform/multimerge+true/_temporary/0/task_local366847577_0019_r_000000
2015-02-12 16:12:50,748 INFO mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) - reduce > reduce
2015-02-12 16:12:50,748 INFO mapred.Task (Task.java:sendDone(1121)) - Task 'attempt_local366847577_0019_r_000000_0' done.
2015-02-12 16:12:50,748 INFO mapred.LocalJobRunner (LocalJobRunner.java:run(325)) - Finishing task: attempt_local366847577_0019_r_000000_0
2015-02-12 16:12:50,749 INFO mapred.LocalJobRunner (LocalJobRunner.java:runTasks(456)) - reduce task executor complete.
2015-02-12 16:12:50,766 INFO util.Hadoop18TapUtil (Hadoop18TapUtil.java:cleanTempPath(219)) - deleting temp path /Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/test/output/hadoop2-mr1/assemblyhelpersplatform/multimerge+true/_temporary
2015-02-12 16:12:50,801 INFO mapred.FileInputFormat (FileInputFormat.java:listStatus(247)) - Total input paths to process : 1
2015-02-12 16:12:50,839 INFO mapred.FileInputFormat (FileInputFormat.java:listStatus(247)) - Total input paths to process : 1
2015-02-12 16:12:50,841 INFO cascading.PlatformTestCase (PlatformTestCase.java:tearDown(209)) - copying to local /Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/test/output//hadoop2-mr1/assemblyhelpersplatform/multimerge+true
2015-02-12 16:12:50,859 INFO util.HadoopUtil (HadoopUtil.java:findMainClass(336)) - using default application jar, may cause class not found exceptions on the cluster
2015-02-12 16:12:50,860 INFO planner.HadoopPlanner (HadoopPlanner.java:initialize(225)) - using application jar: /Users/soberoi/.gradle/caches/modules-2/files-2.1/cascading/cascading-hadoop2-mr1/2.6.3/9ae016e2b8d246b9c1de9168e715d9f1b971aa4a/cascading-hadoop2-mr1-2.6.3.jar
2015-02-12 16:12:50,871 INFO Configuration.deprecation (Configuration.java:warnOnceIfDeprecated(1022)) - mapred.used.genericoptionsparser is deprecated. Instead, use mapreduce.client.genericoptionsparser.used
2015-02-12 16:12:50,873 INFO Configuration.deprecation (Configuration.java:warnOnceIfDeprecated(1022)) - mapred.job.tracker is deprecated. Instead, use mapreduce.jobtracker.address
2015-02-12 16:12:50,878 INFO Configuration.deprecation (Configuration.java:warnOnceIfDeprecated(1022)) - mapred.output.compress is deprecated. Instead, use mapreduce.output.fileoutputformat.compress
2015-02-12 16:12:50,884 INFO flow.Flow (BaseFlow.java:logInfo(1378)) - [second] starting
2015-02-12 16:12:50,885 INFO flow.Flow (BaseFlow.java:logInfo(1378)) - [second] source: Hfs["TextDelimited[['num', 'char']]"]["/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/classes/test/data/lhs.txt"]
2015-02-12 16:12:50,885 INFO flow.Flow (BaseFlow.java:logInfo(1378)) - [second] sink: Hfs["TextDelimited[['char', 'count']]"]["/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/test/output/hadoop2-mr1/assemblyhelpersplatform/samesourcemergecount"]
2015-02-12 16:12:50,885 INFO flow.Flow (BaseFlow.java:logInfo(1378)) - [second] parallel execution is enabled: false
2015-02-12 16:12:50,885 INFO flow.Flow (BaseFlow.java:logInfo(1378)) - [second] starting jobs: 2
2015-02-12 16:12:50,885 INFO flow.Flow (BaseFlow.java:logInfo(1378)) - [second] allocating threads: 1
2015-02-12 16:12:50,885 INFO flow.FlowStep (BaseFlowStep.java:logInfo(834)) - [second] starting step: (1/2)
2015-02-12 16:12:50,886 INFO jvm.JvmMetrics (JvmMetrics.java:init(71)) - Cannot initialize JVM Metrics with processName=JobTracker, sessionId= - already initialized
2015-02-12 16:12:50,887 INFO jvm.JvmMetrics (JvmMetrics.java:init(71)) - Cannot initialize JVM Metrics with processName=JobTracker, sessionId= - already initialized
2015-02-12 16:12:50,908 INFO mapred.FileInputFormat (FileInputFormat.java:listStatus(247)) - Total input paths to process : 1
2015-02-12 16:12:50,924 INFO mapreduce.JobSubmitter (JobSubmitter.java:submitJobInternal(492)) - number of splits:1
2015-02-12 16:12:50,934 INFO mapreduce.JobSubmitter (JobSubmitter.java:printTokens(575)) - Submitting tokens for job: job_local693942087_0020
2015-02-12 16:12:50,943 WARN conf.Configuration (Configuration.java:loadProperty(2510)) - file:/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/tmp/cascading/staging/soberoi693942087/.staging/job_local693942087_0020/job.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.retry.interval; Ignoring.
2015-02-12 16:12:50,944 WARN conf.Configuration (Configuration.java:loadProperty(2510)) - file:/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/tmp/cascading/staging/soberoi693942087/.staging/job_local693942087_0020/job.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.attempts; Ignoring.
2015-02-12 16:12:50,973 WARN conf.Configuration (Configuration.java:loadProperty(2510)) - file:/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/test/tmp/mapred/local/localRunner/soberoi/job_local693942087_0020/job_local693942087_0020.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.retry.interval; Ignoring.
2015-02-12 16:12:50,975 WARN conf.Configuration (Configuration.java:loadProperty(2510)) - file:/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/test/tmp/mapred/local/localRunner/soberoi/job_local693942087_0020/job_local693942087_0020.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.attempts; Ignoring.
2015-02-12 16:12:50,975 INFO mapreduce.Job (Job.java:submit(1299)) - The url to track the job: http://localhost:8080/
2015-02-12 16:12:50,976 INFO mapred.LocalJobRunner (LocalJobRunner.java:createOutputCommitter(471)) - OutputCommitter set in config null
2015-02-12 16:12:50,976 INFO mapred.LocalJobRunner (LocalJobRunner.java:createOutputCommitter(489)) - OutputCommitter is org.apache.hadoop.mapred.FileOutputCommitter
2015-02-12 16:12:50,976 INFO flow.FlowStep (BaseFlowStep.java:logInfo(834)) - [second] submitted hadoop job: job_local693942087_0020
2015-02-12 16:12:50,977 INFO flow.FlowStep (BaseFlowStep.java:logInfo(834)) - [second] tracking url: http://localhost:8080/
2015-02-12 16:12:50,978 INFO mapred.LocalJobRunner (LocalJobRunner.java:runTasks(448)) - Waiting for map tasks
2015-02-12 16:12:50,978 INFO mapred.LocalJobRunner (LocalJobRunner.java:run(224)) - Starting task: attempt_local693942087_0020_m_000000_0
2015-02-12 16:12:50,979 INFO util.ProcfsBasedProcessTree (ProcfsBasedProcessTree.java:isAvailable(181)) - ProcfsBasedProcessTree currently is supported only on Linux.
2015-02-12 16:12:50,979 INFO mapred.Task (Task.java:initialize(587)) - Using ResourceCalculatorProcessTree : null
2015-02-12 16:12:50,980 INFO io.MultiInputSplit (MultiInputSplit.java:readFields(161)) - current split input path: file:/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/classes/test/data/lhs.txt
2015-02-12 16:12:50,980 INFO mapred.MapTask (MapTask.java:updateJobWithSplit(482)) - Processing split: cascading.tap.hadoop.io.MultiInputSplit@4361a2e6
2015-02-12 16:12:50,980 INFO mapred.MapTask (MapTask.java:runOldMapper(436)) - numReduceTasks: 1
2015-02-12 16:12:50,994 INFO mapred.MapTask (MapTask.java:setEquator(1202)) - (EQUATOR) 0 kvi 26214396(104857584)
2015-02-12 16:12:50,994 INFO mapred.MapTask (MapTask.java:init(995)) - mapreduce.task.io.sort.mb: 100
2015-02-12 16:12:50,994 INFO mapred.MapTask (MapTask.java:init(996)) - soft limit at 83886080
2015-02-12 16:12:50,994 INFO mapred.MapTask (MapTask.java:init(997)) - bufstart = 0; bufvoid = 104857600
2015-02-12 16:12:50,995 INFO mapred.MapTask (MapTask.java:init(998)) - kvstart = 26214396; length = 6553600
2015-02-12 16:12:50,996 INFO mapred.MapTask (MapTask.java:createSortingCollector(402)) - Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer
2015-02-12 16:12:50,996 INFO hadoop.FlowMapper (FlowMapper.java:configure(71)) - cascading version: 2.6.3
2015-02-12 16:12:50,996 INFO hadoop.FlowMapper (FlowMapper.java:configure(72)) - child jvm opts: -Xmx200m
2015-02-12 16:12:51,005 INFO hadoop.FlowMapper (FlowMapper.java:configure(87)) - sourcing from: Hfs["TextDelimited[['num', 'char']]"]["/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/classes/test/data/lhs.txt"]
2015-02-12 16:12:51,006 INFO hadoop.FlowMapper (FlowMapper.java:configure(90)) - sinking to: GroupBy(first)[by:[{1}:'char']]
2015-02-12 16:12:51,006 INFO assembly.AggregateBy (AggregateBy.java:prepare(267)) - using threshold value: 2
2015-02-12 16:12:51,009 INFO assembly.AggregateBy (AggregateBy.java:removeEldestEntry(327)) - flushed keys num times: 1, with threshold: 2
2015-02-12 16:12:51,009 INFO assembly.AggregateBy (AggregateBy.java:removeEldestEntry(328)) - mem on flush (mb), free: 155, total: 510, max: 672
2015-02-12 16:12:51,009 INFO assembly.AggregateBy (AggregateBy.java:removeEldestEntry(333)) - total mem is 75% of max mem, to better utilize unused memory consider increasing current LRU threshold with system property "cascading.aggregateby.threshold"
2015-02-12 16:12:51,010 INFO assembly.AggregateBy (AggregateBy.java:removeEldestEntry(327)) - flushed keys num times: 3, with threshold: 2
2015-02-12 16:12:51,010 INFO assembly.AggregateBy (AggregateBy.java:removeEldestEntry(328)) - mem on flush (mb), free: 155, total: 510, max: 672
2015-02-12 16:12:51,010 INFO assembly.AggregateBy (AggregateBy.java:removeEldestEntry(333)) - total mem is 75% of max mem, to better utilize unused memory consider increasing current LRU threshold with system property "cascading.aggregateby.threshold"
2015-02-12 16:12:51,010 INFO assembly.AggregateBy (AggregateBy.java:removeEldestEntry(327)) - flushed keys num times: 5, with threshold: 2
2015-02-12 16:12:51,011 INFO assembly.AggregateBy (AggregateBy.java:removeEldestEntry(328)) - mem on flush (mb), free: 155, total: 510, max: 672
2015-02-12 16:12:51,011 INFO assembly.AggregateBy (AggregateBy.java:removeEldestEntry(333)) - total mem is 75% of max mem, to better utilize unused memory consider increasing current LRU threshold with system property "cascading.aggregateby.threshold"
2015-02-12 16:12:51,011 INFO assembly.AggregateBy (AggregateBy.java:removeEldestEntry(327)) - flushed keys num times: 7, with threshold: 2
2015-02-12 16:12:51,011 INFO assembly.AggregateBy (AggregateBy.java:removeEldestEntry(328)) - mem on flush (mb), free: 155, total: 510, max: 672
2015-02-12 16:12:51,011 INFO assembly.AggregateBy (AggregateBy.java:removeEldestEntry(333)) - total mem is 75% of max mem, to better utilize unused memory consider increasing current LRU threshold with system property "cascading.aggregateby.threshold"
2015-02-12 16:12:51,012 INFO mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) -
2015-02-12 16:12:51,012 INFO mapred.MapTask (MapTask.java:flush(1457)) - Starting flush of map output
2015-02-12 16:12:51,012 INFO mapred.MapTask (MapTask.java:flush(1475)) - Spilling map output
2015-02-12 16:12:51,012 INFO mapred.MapTask (MapTask.java:flush(1476)) - bufstart = 0; bufend = 99; bufvoid = 104857600
2015-02-12 16:12:51,013 INFO mapred.MapTask (MapTask.java:flush(1478)) - kvstart = 26214396(104857584); kvend = 26214364(104857456); length = 33/6553600
2015-02-12 16:12:51,014 INFO mapred.MapTask (MapTask.java:sortAndSpill(1660)) - Finished spill 0
2015-02-12 16:12:51,016 INFO mapred.Task (Task.java:done(1001)) - Task:attempt_local693942087_0020_m_000000_0 is done. And is in the process of committing
2015-02-12 16:12:51,017 INFO mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) - file:/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/classes/test/data/lhs.txt:0+52
2015-02-12 16:12:51,017 INFO mapred.Task (Task.java:sendDone(1121)) - Task 'attempt_local693942087_0020_m_000000_0' done.
2015-02-12 16:12:51,017 INFO mapred.LocalJobRunner (LocalJobRunner.java:run(249)) - Finishing task: attempt_local693942087_0020_m_000000_0
2015-02-12 16:12:51,017 INFO mapred.LocalJobRunner (LocalJobRunner.java:runTasks(456)) - map task executor complete.
2015-02-12 16:12:51,018 INFO mapred.LocalJobRunner (LocalJobRunner.java:runTasks(448)) - Waiting for reduce tasks
2015-02-12 16:12:51,018 INFO mapred.LocalJobRunner (LocalJobRunner.java:run(302)) - Starting task: attempt_local693942087_0020_r_000000_0
2015-02-12 16:12:51,019 INFO util.ProcfsBasedProcessTree (ProcfsBasedProcessTree.java:isAvailable(181)) - ProcfsBasedProcessTree currently is supported only on Linux.
2015-02-12 16:12:51,019 INFO mapred.Task (Task.java:initialize(587)) - Using ResourceCalculatorProcessTree : null
2015-02-12 16:12:51,019 INFO mapred.ReduceTask (ReduceTask.java:run(362)) - Using ShuffleConsumerPlugin: org.apache.hadoop.mapreduce.task.reduce.Shuffle@20155887
2015-02-12 16:12:51,020 INFO reduce.MergeManagerImpl (MergeManagerImpl.java:<init>(195)) - MergerManager: memoryLimit=493250144, maxSingleShuffleLimit=123312536, mergeThreshold=325545120, ioSortFactor=10, memToMemMergeOutputsThreshold=10
2015-02-12 16:12:51,020 INFO reduce.EventFetcher (EventFetcher.java:run(61)) - attempt_local693942087_0020_r_000000_0 Thread started: EventFetcher for fetching Map Completion Events
2015-02-12 16:12:51,021 INFO reduce.LocalFetcher (LocalFetcher.java:copyMapOutput(141)) - localfetcher#16 about to shuffle output of map attempt_local693942087_0020_m_000000_0 decomp: 119 len: 123 to MEMORY
2015-02-12 16:12:51,021 INFO reduce.InMemoryMapOutput (InMemoryMapOutput.java:shuffle(100)) - Read 119 bytes from map-output for attempt_local693942087_0020_m_000000_0
2015-02-12 16:12:51,021 INFO reduce.MergeManagerImpl (MergeManagerImpl.java:closeInMemoryFile(313)) - closeInMemoryFile -> map-output of size: 119, inMemoryMapOutputs.size() -> 1, commitMemory -> 0, usedMemory ->119
2015-02-12 16:12:51,022 INFO reduce.EventFetcher (EventFetcher.java:run(76)) - EventFetcher is interrupted.. Returning
2015-02-12 16:12:51,022 INFO mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) - 1 / 1 copied.
2015-02-12 16:12:51,022 INFO reduce.MergeManagerImpl (MergeManagerImpl.java:finalMerge(673)) - finalMerge called with 1 in-memory map-outputs and 0 on-disk map-outputs
2015-02-12 16:12:51,023 INFO mapred.Merger (Merger.java:merge(606)) - Merging 1 sorted segments
2015-02-12 16:12:51,023 INFO mapred.Merger (Merger.java:merge(705)) - Down to the last merge-pass, with 1 segments left of total size: 110 bytes
2015-02-12 16:12:51,025 INFO reduce.MergeManagerImpl (MergeManagerImpl.java:finalMerge(750)) - Merged 1 segments, 119 bytes to disk to satisfy reduce memory limit
2015-02-12 16:12:51,025 INFO reduce.MergeManagerImpl (MergeManagerImpl.java:finalMerge(780)) - Merging 1 files, 123 bytes from disk
2015-02-12 16:12:51,025 INFO reduce.MergeManagerImpl (MergeManagerImpl.java:finalMerge(795)) - Merging 0 segments, 0 bytes from memory into reduce
2015-02-12 16:12:51,025 INFO mapred.Merger (Merger.java:merge(606)) - Merging 1 sorted segments
2015-02-12 16:12:51,026 INFO mapred.Merger (Merger.java:merge(705)) - Down to the last merge-pass, with 1 segments left of total size: 110 bytes
2015-02-12 16:12:51,026 INFO mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) - 1 / 1 copied.
2015-02-12 16:12:51,027 INFO hadoop.FlowReducer (FlowReducer.java:configure(78)) - cascading version: 2.6.3
2015-02-12 16:12:51,027 INFO hadoop.FlowReducer (FlowReducer.java:configure(79)) - child jvm opts: -Xmx200m
2015-02-12 16:12:51,031 INFO hadoop.FlowReducer (FlowReducer.java:configure(97)) - sourcing from: GroupBy(first)[by:[{1}:'char']]
2015-02-12 16:12:51,032 INFO hadoop.FlowReducer (FlowReducer.java:configure(100)) - sinking to: TempHfs["SequenceFile[['char', 'count']]"][6142022995/first/]
2015-02-12 16:12:51,040 INFO mapred.Task (Task.java:done(1001)) - Task:attempt_local693942087_0020_r_000000_0 is done. And is in the process of committing
2015-02-12 16:12:51,041 INFO mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) - 1 / 1 copied.
2015-02-12 16:12:51,041 INFO mapred.Task (Task.java:commit(1162)) - Task attempt_local693942087_0020_r_000000_0 is allowed to commit now
2015-02-12 16:12:51,041 INFO output.FileOutputCommitter (FileOutputCommitter.java:commitTask(439)) - Saved output of task 'attempt_local693942087_0020_r_000000_0' to file:/tmp/hadoop-soberoi/6142022995_first_F3B3041246EE453FA76367F7FF4B7C5C/_temporary/0/task_local693942087_0020_r_000000
2015-02-12 16:12:51,042 INFO mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) - reduce > reduce
2015-02-12 16:12:51,042 INFO mapred.Task (Task.java:sendDone(1121)) - Task 'attempt_local693942087_0020_r_000000_0' done.
2015-02-12 16:12:51,042 INFO mapred.LocalJobRunner (LocalJobRunner.java:run(325)) - Finishing task: attempt_local693942087_0020_r_000000_0
2015-02-12 16:12:51,043 INFO mapred.LocalJobRunner (LocalJobRunner.java:runTasks(456)) - reduce task executor complete.
2015-02-12 16:12:51,059 INFO flow.FlowStep (BaseFlowStep.java:logInfo(834)) - [second] starting step: (2/2) ...form/samesourcemergecount
2015-02-12 16:12:51,060 INFO jvm.JvmMetrics (JvmMetrics.java:init(71)) - Cannot initialize JVM Metrics with processName=JobTracker, sessionId= - already initialized
2015-02-12 16:12:51,061 INFO jvm.JvmMetrics (JvmMetrics.java:init(71)) - Cannot initialize JVM Metrics with processName=JobTracker, sessionId= - already initialized
2015-02-12 16:12:51,093 INFO mapred.FileInputFormat (FileInputFormat.java:listStatus(247)) - Total input paths to process : 1
2015-02-12 16:12:51,110 INFO mapreduce.JobSubmitter (JobSubmitter.java:submitJobInternal(492)) - number of splits:1
2015-02-12 16:12:51,118 INFO Configuration.deprecation (Configuration.java:warnOnceIfDeprecated(1022)) - mapred.used.genericoptionsparser is deprecated. Instead, use mapreduce.client.genericoptionsparser.used
2015-02-12 16:12:51,118 INFO Configuration.deprecation (Configuration.java:warnOnceIfDeprecated(1022)) - mapred.job.tracker is deprecated. Instead, use mapreduce.jobtracker.address
2015-02-12 16:12:51,121 INFO mapreduce.JobSubmitter (JobSubmitter.java:printTokens(575)) - Submitting tokens for job: job_local782660763_0021
2015-02-12 16:12:51,131 WARN conf.Configuration (Configuration.java:loadProperty(2510)) - file:/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/tmp/cascading/staging/soberoi782660763/.staging/job_local782660763_0021/job.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.retry.interval; Ignoring.
2015-02-12 16:12:51,132 WARN conf.Configuration (Configuration.java:loadProperty(2510)) - file:/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/tmp/cascading/staging/soberoi782660763/.staging/job_local782660763_0021/job.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.attempts; Ignoring.
2015-02-12 16:12:51,160 WARN conf.Configuration (Configuration.java:loadProperty(2510)) - file:/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/test/tmp/mapred/local/localRunner/soberoi/job_local782660763_0021/job_local782660763_0021.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.retry.interval; Ignoring.
2015-02-12 16:12:51,161 WARN conf.Configuration (Configuration.java:loadProperty(2510)) - file:/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/test/tmp/mapred/local/localRunner/soberoi/job_local782660763_0021/job_local782660763_0021.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.attempts; Ignoring.
2015-02-12 16:12:51,161 INFO mapreduce.Job (Job.java:submit(1299)) - The url to track the job: http://localhost:8080/
2015-02-12 16:12:51,162 INFO mapred.LocalJobRunner (LocalJobRunner.java:createOutputCommitter(471)) - OutputCommitter set in config null
2015-02-12 16:12:51,162 INFO flow.FlowStep (BaseFlowStep.java:logInfo(834)) - [second] submitted hadoop job: job_local782660763_0021
2015-02-12 16:12:51,162 INFO mapred.LocalJobRunner (LocalJobRunner.java:createOutputCommitter(489)) - OutputCommitter is org.apache.hadoop.mapred.FileOutputCommitter
2015-02-12 16:12:51,162 INFO flow.FlowStep (BaseFlowStep.java:logInfo(834)) - [second] tracking url: http://localhost:8080/
2015-02-12 16:12:51,163 INFO mapred.LocalJobRunner (LocalJobRunner.java:runTasks(448)) - Waiting for map tasks
2015-02-12 16:12:51,163 INFO mapred.LocalJobRunner (LocalJobRunner.java:run(224)) - Starting task: attempt_local782660763_0021_m_000000_0
2015-02-12 16:12:51,164 INFO util.ProcfsBasedProcessTree (ProcfsBasedProcessTree.java:isAvailable(181)) - ProcfsBasedProcessTree currently is supported only on Linux.
2015-02-12 16:12:51,164 INFO mapred.Task (Task.java:initialize(587)) - Using ResourceCalculatorProcessTree : null
2015-02-12 16:12:51,165 INFO io.MultiInputSplit (MultiInputSplit.java:readFields(161)) - current split input path: file:/tmp/hadoop-soberoi/6142022995_first_F3B3041246EE453FA76367F7FF4B7C5C/part-00000
2015-02-12 16:12:51,165 INFO mapred.MapTask (MapTask.java:updateJobWithSplit(482)) - Processing split: cascading.tap.hadoop.io.MultiInputSplit@6e9e36f1
2015-02-12 16:12:51,166 INFO mapred.MapTask (MapTask.java:runOldMapper(436)) - numReduceTasks: 1
2015-02-12 16:12:51,181 INFO mapred.MapTask (MapTask.java:setEquator(1202)) - (EQUATOR) 0 kvi 26214396(104857584)
2015-02-12 16:12:51,181 INFO mapred.MapTask (MapTask.java:init(995)) - mapreduce.task.io.sort.mb: 100
2015-02-12 16:12:51,181 INFO mapred.MapTask (MapTask.java:init(996)) - soft limit at 83886080
2015-02-12 16:12:51,181 INFO mapred.MapTask (MapTask.java:init(997)) - bufstart = 0; bufvoid = 104857600
2015-02-12 16:12:51,181 INFO mapred.MapTask (MapTask.java:init(998)) - kvstart = 26214396; length = 6553600
2015-02-12 16:12:51,182 INFO mapred.MapTask (MapTask.java:createSortingCollector(402)) - Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer
2015-02-12 16:12:51,182 INFO hadoop.FlowMapper (FlowMapper.java:configure(71)) - cascading version: 2.6.3
2015-02-12 16:12:51,182 INFO hadoop.FlowMapper (FlowMapper.java:configure(72)) - child jvm opts: -Xmx200m
2015-02-12 16:12:51,187 INFO hadoop.FlowMapper (FlowMapper.java:configure(87)) - sourcing from: TempHfs["SequenceFile[['char', 'count']]"][6142022995/first/]
2015-02-12 16:12:51,187 INFO hadoop.FlowMapper (FlowMapper.java:configure(90)) - sinking to: GroupBy(second)[by:[{1}:'char']]
2015-02-12 16:12:51,188 INFO assembly.AggregateBy (AggregateBy.java:prepare(267)) - using threshold value: 2
2015-02-12 16:12:51,189 INFO assembly.AggregateBy (AggregateBy.java:removeEldestEntry(327)) - flushed keys num times: 1, with threshold: 2
2015-02-12 16:12:51,189 INFO assembly.AggregateBy (AggregateBy.java:removeEldestEntry(328)) - mem on flush (mb), free: 196, total: 510, max: 672
2015-02-12 16:12:51,189 INFO assembly.AggregateBy (AggregateBy.java:removeEldestEntry(333)) - total mem is 75% of max mem, to better utilize unused memory consider increasing current LRU threshold with system property "cascading.aggregateby.threshold"
2015-02-12 16:12:51,189 INFO assembly.AggregateBy (AggregateBy.java:removeEldestEntry(327)) - flushed keys num times: 3, with threshold: 2
2015-02-12 16:12:51,189 INFO assembly.AggregateBy (AggregateBy.java:removeEldestEntry(328)) - mem on flush (mb), free: 196, total: 510, max: 672
2015-02-12 16:12:51,189 INFO assembly.AggregateBy (AggregateBy.java:removeEldestEntry(333)) - total mem is 75% of max mem, to better utilize unused memory consider increasing current LRU threshold with system property "cascading.aggregateby.threshold"
2015-02-12 16:12:51,191 INFO mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) -
2015-02-12 16:12:51,191 INFO mapred.MapTask (MapTask.java:flush(1457)) - Starting flush of map output
2015-02-12 16:12:51,191 INFO mapred.MapTask (MapTask.java:flush(1475)) - Spilling map output
2015-02-12 16:12:51,191 INFO mapred.MapTask (MapTask.java:flush(1476)) - bufstart = 0; bufend = 55; bufvoid = 104857600
2015-02-12 16:12:51,191 INFO mapred.MapTask (MapTask.java:flush(1478)) - kvstart = 26214396(104857584); kvend = 26214380(104857520); length = 17/6553600
2015-02-12 16:12:51,193 INFO mapred.MapTask (MapTask.java:sortAndSpill(1660)) - Finished spill 0
2015-02-12 16:12:51,194 INFO mapred.Task (Task.java:done(1001)) - Task:attempt_local782660763_0021_m_000000_0 is done. And is in the process of committing
2015-02-12 16:12:51,195 INFO mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) - file:/tmp/hadoop-soberoi/6142022995_first_F3B3041246EE453FA76367F7FF4B7C5C/part-00000:0+160
2015-02-12 16:12:51,195 INFO mapred.Task (Task.java:sendDone(1121)) - Task 'attempt_local782660763_0021_m_000000_0' done.
2015-02-12 16:12:51,195 INFO mapred.LocalJobRunner (LocalJobRunner.java:run(249)) - Finishing task: attempt_local782660763_0021_m_000000_0
2015-02-12 16:12:51,195 INFO mapred.LocalJobRunner (LocalJobRunner.java:runTasks(456)) - map task executor complete.
2015-02-12 16:12:51,195 INFO mapred.LocalJobRunner (LocalJobRunner.java:runTasks(448)) - Waiting for reduce tasks
2015-02-12 16:12:51,195 INFO mapred.LocalJobRunner (LocalJobRunner.java:run(302)) - Starting task: attempt_local782660763_0021_r_000000_0
2015-02-12 16:12:51,196 INFO util.ProcfsBasedProcessTree (ProcfsBasedProcessTree.java:isAvailable(181)) - ProcfsBasedProcessTree currently is supported only on Linux.
2015-02-12 16:12:51,197 INFO mapred.Task (Task.java:initialize(587)) - Using ResourceCalculatorProcessTree : null
2015-02-12 16:12:51,197 INFO mapred.ReduceTask (ReduceTask.java:run(362)) - Using ShuffleConsumerPlugin: org.apache.hadoop.mapreduce.task.reduce.Shuffle@67ea92f3
2015-02-12 16:12:51,197 INFO reduce.MergeManagerImpl (MergeManagerImpl.java:<init>(195)) - MergerManager: memoryLimit=493250144, maxSingleShuffleLimit=123312536, mergeThreshold=325545120, ioSortFactor=10, memToMemMergeOutputsThreshold=10
2015-02-12 16:12:51,197 INFO reduce.EventFetcher (EventFetcher.java:run(61)) - attempt_local782660763_0021_r_000000_0 Thread started: EventFetcher for fetching Map Completion Events
2015-02-12 16:12:51,198 INFO reduce.LocalFetcher (LocalFetcher.java:copyMapOutput(141)) - localfetcher#17 about to shuffle output of map attempt_local782660763_0021_m_000000_0 decomp: 67 len: 71 to MEMORY
2015-02-12 16:12:51,198 INFO reduce.InMemoryMapOutput (InMemoryMapOutput.java:shuffle(100)) - Read 67 bytes from map-output for attempt_local782660763_0021_m_000000_0
2015-02-12 16:12:51,199 INFO reduce.MergeManagerImpl (MergeManagerImpl.java:closeInMemoryFile(313)) - closeInMemoryFile -> map-output of size: 67, inMemoryMapOutputs.size() -> 1, commitMemory -> 0, usedMemory ->67
2015-02-12 16:12:51,199 INFO reduce.EventFetcher (EventFetcher.java:run(76)) - EventFetcher is interrupted.. Returning
2015-02-12 16:12:51,199 INFO mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) - 1 / 1 copied.
2015-02-12 16:12:51,200 INFO reduce.MergeManagerImpl (MergeManagerImpl.java:finalMerge(673)) - finalMerge called with 1 in-memory map-outputs and 0 on-disk map-outputs
2015-02-12 16:12:51,201 INFO mapred.Merger (Merger.java:merge(606)) - Merging 1 sorted segments
2015-02-12 16:12:51,201 INFO mapred.Merger (Merger.java:merge(705)) - Down to the last merge-pass, with 1 segments left of total size: 58 bytes
2015-02-12 16:12:51,202 INFO reduce.MergeManagerImpl (MergeManagerImpl.java:finalMerge(750)) - Merged 1 segments, 67 bytes to disk to satisfy reduce memory limit
2015-02-12 16:12:51,202 INFO reduce.MergeManagerImpl (MergeManagerImpl.java:finalMerge(780)) - Merging 1 files, 71 bytes from disk
2015-02-12 16:12:51,202 INFO reduce.MergeManagerImpl (MergeManagerImpl.java:finalMerge(795)) - Merging 0 segments, 0 bytes from memory into reduce
2015-02-12 16:12:51,202 INFO mapred.Merger (Merger.java:merge(606)) - Merging 1 sorted segments
2015-02-12 16:12:51,202 INFO mapred.Merger (Merger.java:merge(705)) - Down to the last merge-pass, with 1 segments left of total size: 58 bytes
2015-02-12 16:12:51,203 INFO mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) - 1 / 1 copied.
2015-02-12 16:12:51,203 INFO hadoop.FlowReducer (FlowReducer.java:configure(78)) - cascading version: 2.6.3
2015-02-12 16:12:51,203 INFO hadoop.FlowReducer (FlowReducer.java:configure(79)) - child jvm opts: -Xmx200m
2015-02-12 16:12:51,209 INFO hadoop.FlowReducer (FlowReducer.java:configure(97)) - sourcing from: GroupBy(second)[by:[{1}:'char']]
2015-02-12 16:12:51,209 INFO hadoop.FlowReducer (FlowReducer.java:configure(100)) - sinking to: Hfs["TextDelimited[['char', 'count']]"]["/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/test/output/hadoop2-mr1/assemblyhelpersplatform/samesourcemergecount"]
2015-02-12 16:12:51,217 INFO mapred.Task (Task.java:done(1001)) - Task:attempt_local782660763_0021_r_000000_0 is done. And is in the process of committing
2015-02-12 16:12:51,218 INFO mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) - 1 / 1 copied.
2015-02-12 16:12:51,218 INFO mapred.Task (Task.java:commit(1162)) - Task attempt_local782660763_0021_r_000000_0 is allowed to commit now
2015-02-12 16:12:51,219 INFO output.FileOutputCommitter (FileOutputCommitter.java:commitTask(439)) - Saved output of task 'attempt_local782660763_0021_r_000000_0' to file:/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/test/output/hadoop2-mr1/assemblyhelpersplatform/samesourcemergecount/_temporary/0/task_local782660763_0021_r_000000
2015-02-12 16:12:51,219 INFO mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) - reduce > reduce
2015-02-12 16:12:51,219 INFO mapred.Task (Task.java:sendDone(1121)) - Task 'attempt_local782660763_0021_r_000000_0' done.
2015-02-12 16:12:51,220 INFO mapred.LocalJobRunner (LocalJobRunner.java:run(325)) - Finishing task: attempt_local782660763_0021_r_000000_0
2015-02-12 16:12:51,220 INFO mapred.LocalJobRunner (LocalJobRunner.java:runTasks(456)) - reduce task executor complete.
2015-02-12 16:12:51,238 INFO util.Hadoop18TapUtil (Hadoop18TapUtil.java:cleanTempPath(219)) - deleting temp path /Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/test/output/hadoop2-mr1/assemblyhelpersplatform/samesourcemergecount/_temporary
2015-02-12 16:12:51,257 INFO mapred.FileInputFormat (FileInputFormat.java:listStatus(247)) - Total input paths to process : 1
2015-02-12 16:12:51,278 INFO mapred.FileInputFormat (FileInputFormat.java:listStatus(247)) - Total input paths to process : 1
2015-02-12 16:12:51,279 INFO cascading.PlatformTestCase (PlatformTestCase.java:tearDown(209)) - copying to local /Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/test/output//hadoop2-mr1/assemblyhelpersplatform/samesourcemergecount
2015-02-12 16:12:51,290 INFO util.HadoopUtil (HadoopUtil.java:findMainClass(336)) - using default application jar, may cause class not found exceptions on the cluster
2015-02-12 16:12:51,290 INFO planner.HadoopPlanner (HadoopPlanner.java:initialize(225)) - using application jar: /Users/soberoi/.gradle/caches/modules-2/files-2.1/cascading/cascading-hadoop2-mr1/2.6.3/9ae016e2b8d246b9c1de9168e715d9f1b971aa4a/cascading-hadoop2-mr1-2.6.3.jar
2015-02-12 16:12:51,293 INFO Configuration.deprecation (Configuration.java:warnOnceIfDeprecated(1022)) - mapred.used.genericoptionsparser is deprecated. Instead, use mapreduce.client.genericoptionsparser.used
2015-02-12 16:12:51,294 INFO Configuration.deprecation (Configuration.java:warnOnceIfDeprecated(1022)) - mapred.job.tracker is deprecated. Instead, use mapreduce.jobtracker.address
2015-02-12 16:12:51,294 INFO Configuration.deprecation (Configuration.java:warnOnceIfDeprecated(1022)) - mapred.output.compress is deprecated. Instead, use mapreduce.output.fileoutputformat.compress
2015-02-12 16:12:51,298 INFO flow.Flow (BaseFlow.java:logInfo(1378)) - [average] starting
2015-02-12 16:12:51,298 INFO flow.Flow (BaseFlow.java:logInfo(1378)) - [average] source: Hfs["TextDelimited[['num', 'char']]"]["/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/classes/test/data/lhs.txt"]
2015-02-12 16:12:51,298 INFO flow.Flow (BaseFlow.java:logInfo(1378)) - [average] sink: Hfs["TextDelimited[['char', 'average']]"]["/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/test/output/hadoop2-mr1/assemblyhelpersplatform/average"]
2015-02-12 16:12:51,298 INFO flow.Flow (BaseFlow.java:logInfo(1378)) - [average] parallel execution is enabled: false
2015-02-12 16:12:51,298 INFO flow.Flow (BaseFlow.java:logInfo(1378)) - [average] starting jobs: 1
2015-02-12 16:12:51,298 INFO flow.Flow (BaseFlow.java:logInfo(1378)) - [average] allocating threads: 1
2015-02-12 16:12:51,299 INFO flow.FlowStep (BaseFlowStep.java:logInfo(834)) - [average] starting step: (1/1) ...lyhelpersplatform/average
2015-02-12 16:12:51,299 INFO jvm.JvmMetrics (JvmMetrics.java:init(71)) - Cannot initialize JVM Metrics with processName=JobTracker, sessionId= - already initialized
2015-02-12 16:12:51,300 INFO jvm.JvmMetrics (JvmMetrics.java:init(71)) - Cannot initialize JVM Metrics with processName=JobTracker, sessionId= - already initialized
2015-02-12 16:12:51,319 INFO mapred.FileInputFormat (FileInputFormat.java:listStatus(247)) - Total input paths to process : 1
2015-02-12 16:12:51,334 INFO mapreduce.JobSubmitter (JobSubmitter.java:submitJobInternal(492)) - number of splits:1
2015-02-12 16:12:51,345 INFO mapreduce.JobSubmitter (JobSubmitter.java:printTokens(575)) - Submitting tokens for job: job_local1315145036_0022
2015-02-12 16:12:51,358 WARN conf.Configuration (Configuration.java:loadProperty(2510)) - file:/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/tmp/cascading/staging/soberoi1315145036/.staging/job_local1315145036_0022/job.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.retry.interval; Ignoring.
2015-02-12 16:12:51,359 WARN conf.Configuration (Configuration.java:loadProperty(2510)) - file:/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/tmp/cascading/staging/soberoi1315145036/.staging/job_local1315145036_0022/job.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.attempts; Ignoring.
2015-02-12 16:12:51,389 WARN conf.Configuration (Configuration.java:loadProperty(2510)) - file:/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/test/tmp/mapred/local/localRunner/soberoi/job_local1315145036_0022/job_local1315145036_0022.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.retry.interval; Ignoring.
2015-02-12 16:12:51,390 WARN conf.Configuration (Configuration.java:loadProperty(2510)) - file:/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/test/tmp/mapred/local/localRunner/soberoi/job_local1315145036_0022/job_local1315145036_0022.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.attempts; Ignoring.
2015-02-12 16:12:51,391 INFO mapreduce.Job (Job.java:submit(1299)) - The url to track the job: http://localhost:8080/
2015-02-12 16:12:51,391 INFO flow.FlowStep (BaseFlowStep.java:logInfo(834)) - [average] submitted hadoop job: job_local1315145036_0022
2015-02-12 16:12:51,391 INFO mapred.LocalJobRunner (LocalJobRunner.java:createOutputCommitter(471)) - OutputCommitter set in config null
2015-02-12 16:12:51,391 INFO flow.FlowStep (BaseFlowStep.java:logInfo(834)) - [average] tracking url: http://localhost:8080/
2015-02-12 16:12:51,391 INFO mapred.LocalJobRunner (LocalJobRunner.java:createOutputCommitter(489)) - OutputCommitter is org.apache.hadoop.mapred.FileOutputCommitter
2015-02-12 16:12:51,393 INFO mapred.LocalJobRunner (LocalJobRunner.java:runTasks(448)) - Waiting for map tasks
2015-02-12 16:12:51,393 INFO mapred.LocalJobRunner (LocalJobRunner.java:run(224)) - Starting task: attempt_local1315145036_0022_m_000000_0
2015-02-12 16:12:51,394 INFO util.ProcfsBasedProcessTree (ProcfsBasedProcessTree.java:isAvailable(181)) - ProcfsBasedProcessTree currently is supported only on Linux.
2015-02-12 16:12:51,394 INFO mapred.Task (Task.java:initialize(587)) - Using ResourceCalculatorProcessTree : null
2015-02-12 16:12:51,394 INFO io.MultiInputSplit (MultiInputSplit.java:readFields(161)) - current split input path: file:/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/classes/test/data/lhs.txt
2015-02-12 16:12:51,395 INFO mapred.MapTask (MapTask.java:updateJobWithSplit(482)) - Processing split: cascading.tap.hadoop.io.MultiInputSplit@41b2fd97
2015-02-12 16:12:51,395 INFO mapred.MapTask (MapTask.java:runOldMapper(436)) - numReduceTasks: 1
2015-02-12 16:12:51,410 INFO mapred.MapTask (MapTask.java:setEquator(1202)) - (EQUATOR) 0 kvi 26214396(104857584)
2015-02-12 16:12:51,410 INFO mapred.MapTask (MapTask.java:init(995)) - mapreduce.task.io.sort.mb: 100
2015-02-12 16:12:51,410 INFO mapred.MapTask (MapTask.java:init(996)) - soft limit at 83886080
2015-02-12 16:12:51,410 INFO mapred.MapTask (MapTask.java:init(997)) - bufstart = 0; bufvoid = 104857600
2015-02-12 16:12:51,410 INFO mapred.MapTask (MapTask.java:init(998)) - kvstart = 26214396; length = 6553600
2015-02-12 16:12:51,411 INFO mapred.MapTask (MapTask.java:createSortingCollector(402)) - Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer
2015-02-12 16:12:51,411 INFO hadoop.FlowMapper (FlowMapper.java:configure(71)) - cascading version: 2.6.3
2015-02-12 16:12:51,411 INFO hadoop.FlowMapper (FlowMapper.java:configure(72)) - child jvm opts: -Xmx200m
2015-02-12 16:12:51,415 INFO hadoop.FlowMapper (FlowMapper.java:configure(87)) - sourcing from: Hfs["TextDelimited[['num', 'char']]"]["/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/classes/test/data/lhs.txt"]
2015-02-12 16:12:51,415 INFO hadoop.FlowMapper (FlowMapper.java:configure(90)) - sinking to: GroupBy(average)[by:[{1}:'char']]
2015-02-12 16:12:51,415 INFO assembly.AggregateBy (AggregateBy.java:prepare(267)) - using threshold value: 2
2015-02-12 16:12:51,416 INFO assembly.AggregateBy (AggregateBy.java:removeEldestEntry(327)) - flushed keys num times: 1, with threshold: 2
2015-02-12 16:12:51,416 INFO assembly.AggregateBy (AggregateBy.java:removeEldestEntry(328)) - mem on flush (mb), free: 197, total: 511, max: 672
2015-02-12 16:12:51,416 INFO assembly.AggregateBy (AggregateBy.java:removeEldestEntry(333)) - total mem is 76% of max mem, to better utilize unused memory consider increasing current LRU threshold with system property "cascading.aggregateby.threshold"
2015-02-12 16:12:51,417 INFO assembly.AggregateBy (AggregateBy.java:removeEldestEntry(327)) - flushed keys num times: 3, with threshold: 2
2015-02-12 16:12:51,417 INFO assembly.AggregateBy (AggregateBy.java:removeEldestEntry(328)) - mem on flush (mb), free: 197, total: 511, max: 672
2015-02-12 16:12:51,417 INFO assembly.AggregateBy (AggregateBy.java:removeEldestEntry(333)) - total mem is 76% of max mem, to better utilize unused memory consider increasing current LRU threshold with system property "cascading.aggregateby.threshold"
2015-02-12 16:12:51,417 INFO assembly.AggregateBy (AggregateBy.java:removeEldestEntry(327)) - flushed keys num times: 5, with threshold: 2
2015-02-12 16:12:51,418 INFO assembly.AggregateBy (AggregateBy.java:removeEldestEntry(328)) - mem on flush (mb), free: 196, total: 511, max: 672
2015-02-12 16:12:51,418 INFO assembly.AggregateBy (AggregateBy.java:removeEldestEntry(333)) - total mem is 76% of max mem, to better utilize unused memory consider increasing current LRU threshold with system property "cascading.aggregateby.threshold"
2015-02-12 16:12:51,418 INFO assembly.AggregateBy (AggregateBy.java:removeEldestEntry(327)) - flushed keys num times: 7, with threshold: 2
2015-02-12 16:12:51,418 INFO assembly.AggregateBy (AggregateBy.java:removeEldestEntry(328)) - mem on flush (mb), free: 196, total: 511, max: 672
2015-02-12 16:12:51,418 INFO assembly.AggregateBy (AggregateBy.java:removeEldestEntry(333)) - total mem is 76% of max mem, to better utilize unused memory consider increasing current LRU threshold with system property "cascading.aggregateby.threshold"
2015-02-12 16:12:51,419 INFO mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) -
2015-02-12 16:12:51,420 INFO mapred.MapTask (MapTask.java:flush(1457)) - Starting flush of map output
2015-02-12 16:12:51,420 INFO mapred.MapTask (MapTask.java:flush(1475)) - Spilling map output
2015-02-12 16:12:51,420 INFO mapred.MapTask (MapTask.java:flush(1476)) - bufstart = 0; bufend = 180; bufvoid = 104857600
2015-02-12 16:12:51,420 INFO mapred.MapTask (MapTask.java:flush(1478)) - kvstart = 26214396(104857584); kvend = 26214364(104857456); length = 33/6553600
2015-02-12 16:12:51,422 INFO mapred.MapTask (MapTask.java:sortAndSpill(1660)) - Finished spill 0
2015-02-12 16:12:51,423 INFO mapred.Task (Task.java:done(1001)) - Task:attempt_local1315145036_0022_m_000000_0 is done. And is in the process of committing
2015-02-12 16:12:51,424 INFO mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) - file:/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/classes/test/data/lhs.txt:0+52
2015-02-12 16:12:51,424 INFO mapred.Task (Task.java:sendDone(1121)) - Task 'attempt_local1315145036_0022_m_000000_0' done.
2015-02-12 16:12:51,424 INFO mapred.LocalJobRunner (LocalJobRunner.java:run(249)) - Finishing task: attempt_local1315145036_0022_m_000000_0
2015-02-12 16:12:51,424 INFO mapred.LocalJobRunner (LocalJobRunner.java:runTasks(456)) - map task executor complete.
2015-02-12 16:12:51,425 INFO mapred.LocalJobRunner (LocalJobRunner.java:runTasks(448)) - Waiting for reduce tasks
2015-02-12 16:12:51,425 INFO mapred.LocalJobRunner (LocalJobRunner.java:run(302)) - Starting task: attempt_local1315145036_0022_r_000000_0
2015-02-12 16:12:51,426 INFO util.ProcfsBasedProcessTree (ProcfsBasedProcessTree.java:isAvailable(181)) - ProcfsBasedProcessTree currently is supported only on Linux.
2015-02-12 16:12:51,426 INFO mapred.Task (Task.java:initialize(587)) - Using ResourceCalculatorProcessTree : null
2015-02-12 16:12:51,426 INFO mapred.ReduceTask (ReduceTask.java:run(362)) - Using ShuffleConsumerPlugin: org.apache.hadoop.mapreduce.task.reduce.Shuffle@7f317c99
2015-02-12 16:12:51,427 INFO reduce.MergeManagerImpl (MergeManagerImpl.java:<init>(195)) - MergerManager: memoryLimit=493250144, maxSingleShuffleLimit=123312536, mergeThreshold=325545120, ioSortFactor=10, memToMemMergeOutputsThreshold=10
2015-02-12 16:12:51,428 INFO reduce.EventFetcher (EventFetcher.java:run(61)) - attempt_local1315145036_0022_r_000000_0 Thread started: EventFetcher for fetching Map Completion Events
2015-02-12 16:12:51,428 INFO reduce.LocalFetcher (LocalFetcher.java:copyMapOutput(141)) - localfetcher#18 about to shuffle output of map attempt_local1315145036_0022_m_000000_0 decomp: 200 len: 204 to MEMORY
2015-02-12 16:12:51,429 INFO reduce.InMemoryMapOutput (InMemoryMapOutput.java:shuffle(100)) - Read 200 bytes from map-output for attempt_local1315145036_0022_m_000000_0
2015-02-12 16:12:51,429 INFO reduce.MergeManagerImpl (MergeManagerImpl.java:closeInMemoryFile(313)) - closeInMemoryFile -> map-output of size: 200, inMemoryMapOutputs.size() -> 1, commitMemory -> 0, usedMemory ->200
2015-02-12 16:12:51,429 INFO reduce.EventFetcher (EventFetcher.java:run(76)) - EventFetcher is interrupted.. Returning
2015-02-12 16:12:51,430 INFO mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) - 1 / 1 copied.
2015-02-12 16:12:51,430 INFO reduce.MergeManagerImpl (MergeManagerImpl.java:finalMerge(673)) - finalMerge called with 1 in-memory map-outputs and 0 on-disk map-outputs
2015-02-12 16:12:51,431 INFO mapred.Merger (Merger.java:merge(606)) - Merging 1 sorted segments
2015-02-12 16:12:51,431 INFO mapred.Merger (Merger.java:merge(705)) - Down to the last merge-pass, with 1 segments left of total size: 191 bytes
2015-02-12 16:12:51,432 INFO reduce.MergeManagerImpl (MergeManagerImpl.java:finalMerge(750)) - Merged 1 segments, 200 bytes to disk to satisfy reduce memory limit
2015-02-12 16:12:51,433 INFO reduce.MergeManagerImpl (MergeManagerImpl.java:finalMerge(780)) - Merging 1 files, 204 bytes from disk
2015-02-12 16:12:51,433 INFO reduce.MergeManagerImpl (MergeManagerImpl.java:finalMerge(795)) - Merging 0 segments, 0 bytes from memory into reduce
2015-02-12 16:12:51,433 INFO mapred.Merger (Merger.java:merge(606)) - Merging 1 sorted segments
2015-02-12 16:12:51,433 INFO mapred.Merger (Merger.java:merge(705)) - Down to the last merge-pass, with 1 segments left of total size: 191 bytes
2015-02-12 16:12:51,433 INFO mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) - 1 / 1 copied.
2015-02-12 16:12:51,434 INFO hadoop.FlowReducer (FlowReducer.java:configure(78)) - cascading version: 2.6.3
2015-02-12 16:12:51,434 INFO hadoop.FlowReducer (FlowReducer.java:configure(79)) - child jvm opts: -Xmx200m
2015-02-12 16:12:51,438 INFO hadoop.FlowReducer (FlowReducer.java:configure(97)) - sourcing from: GroupBy(average)[by:[{1}:'char']]
2015-02-12 16:12:51,439 INFO hadoop.FlowReducer (FlowReducer.java:configure(100)) - sinking to: Hfs["TextDelimited[['char', 'average']]"]["/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/test/output/hadoop2-mr1/assemblyhelpersplatform/average"]
2015-02-12 16:12:51,447 INFO mapred.Task (Task.java:done(1001)) - Task:attempt_local1315145036_0022_r_000000_0 is done. And is in the process of committing
2015-02-12 16:12:51,448 INFO mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) - 1 / 1 copied.
2015-02-12 16:12:51,448 INFO mapred.Task (Task.java:commit(1162)) - Task attempt_local1315145036_0022_r_000000_0 is allowed to commit now
2015-02-12 16:12:51,449 INFO output.FileOutputCommitter (FileOutputCommitter.java:commitTask(439)) - Saved output of task 'attempt_local1315145036_0022_r_000000_0' to file:/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/test/output/hadoop2-mr1/assemblyhelpersplatform/average/_temporary/0/task_local1315145036_0022_r_000000
2015-02-12 16:12:51,450 INFO mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) - reduce > reduce
2015-02-12 16:12:51,450 INFO mapred.Task (Task.java:sendDone(1121)) - Task 'attempt_local1315145036_0022_r_000000_0' done.
2015-02-12 16:12:51,450 INFO mapred.LocalJobRunner (LocalJobRunner.java:run(325)) - Finishing task: attempt_local1315145036_0022_r_000000_0
2015-02-12 16:12:51,450 INFO mapred.LocalJobRunner (LocalJobRunner.java:runTasks(456)) - reduce task executor complete.
2015-02-12 16:12:51,461 INFO util.Hadoop18TapUtil (Hadoop18TapUtil.java:cleanTempPath(219)) - deleting temp path /Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/test/output/hadoop2-mr1/assemblyhelpersplatform/average/_temporary
2015-02-12 16:12:51,487 INFO mapred.FileInputFormat (FileInputFormat.java:listStatus(247)) - Total input paths to process : 1
2015-02-12 16:12:51,507 INFO mapred.FileInputFormat (FileInputFormat.java:listStatus(247)) - Total input paths to process : 1
2015-02-12 16:12:51,508 INFO cascading.PlatformTestCase (PlatformTestCase.java:tearDown(209)) - copying to local /Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/test/output//hadoop2-mr1/assemblyhelpersplatform/average
2015-02-12 16:12:51,518 INFO util.HadoopUtil (HadoopUtil.java:findMainClass(336)) - using default application jar, may cause class not found exceptions on the cluster
2015-02-12 16:12:51,518 INFO planner.HadoopPlanner (HadoopPlanner.java:initialize(225)) - using application jar: /Users/soberoi/.gradle/caches/modules-2/files-2.1/cascading/cascading-hadoop2-mr1/2.6.3/9ae016e2b8d246b9c1de9168e715d9f1b971aa4a/cascading-hadoop2-mr1-2.6.3.jar
2015-02-12 16:12:51,522 INFO Configuration.deprecation (Configuration.java:warnOnceIfDeprecated(1022)) - mapred.used.genericoptionsparser is deprecated. Instead, use mapreduce.client.genericoptionsparser.used
2015-02-12 16:12:51,523 INFO Configuration.deprecation (Configuration.java:warnOnceIfDeprecated(1022)) - mapred.job.tracker is deprecated. Instead, use mapreduce.jobtracker.address
2015-02-12 16:12:51,524 INFO Configuration.deprecation (Configuration.java:warnOnceIfDeprecated(1022)) - mapred.output.compress is deprecated. Instead, use mapreduce.output.fileoutputformat.compress
2015-02-12 16:12:51,528 INFO flow.Flow (BaseFlow.java:logInfo(1378)) - [max] starting
2015-02-12 16:12:51,528 INFO flow.Flow (BaseFlow.java:logInfo(1378)) - [max] source: Hfs["TextDelimited[['num', 'char']]"]["/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/classes/test/data/lhs.txt"]
2015-02-12 16:12:51,528 INFO flow.Flow (BaseFlow.java:logInfo(1378)) - [max] sink: Hfs["TextDelimited[['char', 'max']]"]["/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/test/output/hadoop2-mr1/assemblyhelpersplatform/maxbynullsafety"]
2015-02-12 16:12:51,529 INFO flow.Flow (BaseFlow.java:logInfo(1378)) - [max] parallel execution is enabled: false
2015-02-12 16:12:51,529 INFO flow.Flow (BaseFlow.java:logInfo(1378)) - [max] starting jobs: 1
2015-02-12 16:12:51,529 INFO flow.Flow (BaseFlow.java:logInfo(1378)) - [max] allocating threads: 1
2015-02-12 16:12:51,529 INFO flow.FlowStep (BaseFlowStep.java:logInfo(834)) - [max] starting step: (1/1) ...splatform/maxbynullsafety
2015-02-12 16:12:51,530 INFO jvm.JvmMetrics (JvmMetrics.java:init(71)) - Cannot initialize JVM Metrics with processName=JobTracker, sessionId= - already initialized
2015-02-12 16:12:51,530 INFO jvm.JvmMetrics (JvmMetrics.java:init(71)) - Cannot initialize JVM Metrics with processName=JobTracker, sessionId= - already initialized
2015-02-12 16:12:51,547 INFO mapred.FileInputFormat (FileInputFormat.java:listStatus(247)) - Total input paths to process : 1
2015-02-12 16:12:51,563 INFO mapreduce.JobSubmitter (JobSubmitter.java:submitJobInternal(492)) - number of splits:1
2015-02-12 16:12:51,575 INFO mapreduce.JobSubmitter (JobSubmitter.java:printTokens(575)) - Submitting tokens for job: job_local1388077210_0023
2015-02-12 16:12:51,587 WARN conf.Configuration (Configuration.java:loadProperty(2510)) - file:/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/tmp/cascading/staging/soberoi1388077210/.staging/job_local1388077210_0023/job.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.retry.interval; Ignoring.
2015-02-12 16:12:51,587 WARN conf.Configuration (Configuration.java:loadProperty(2510)) - file:/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/tmp/cascading/staging/soberoi1388077210/.staging/job_local1388077210_0023/job.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.attempts; Ignoring.
2015-02-12 16:12:51,614 WARN conf.Configuration (Configuration.java:loadProperty(2510)) - file:/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/test/tmp/mapred/local/localRunner/soberoi/job_local1388077210_0023/job_local1388077210_0023.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.retry.interval; Ignoring.
2015-02-12 16:12:51,615 WARN conf.Configuration (Configuration.java:loadProperty(2510)) - file:/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/test/tmp/mapred/local/localRunner/soberoi/job_local1388077210_0023/job_local1388077210_0023.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.attempts; Ignoring.
2015-02-12 16:12:51,615 INFO mapreduce.Job (Job.java:submit(1299)) - The url to track the job: http://localhost:8080/
2015-02-12 16:12:51,615 INFO mapred.LocalJobRunner (LocalJobRunner.java:createOutputCommitter(471)) - OutputCommitter set in config null
2015-02-12 16:12:51,615 INFO flow.FlowStep (BaseFlowStep.java:logInfo(834)) - [max] submitted hadoop job: job_local1388077210_0023
2015-02-12 16:12:51,616 INFO flow.FlowStep (BaseFlowStep.java:logInfo(834)) - [max] tracking url: http://localhost:8080/
2015-02-12 16:12:51,616 INFO mapred.LocalJobRunner (LocalJobRunner.java:createOutputCommitter(489)) - OutputCommitter is org.apache.hadoop.mapred.FileOutputCommitter
2015-02-12 16:12:51,617 INFO mapred.LocalJobRunner (LocalJobRunner.java:runTasks(448)) - Waiting for map tasks
2015-02-12 16:12:51,617 INFO mapred.LocalJobRunner (LocalJobRunner.java:run(224)) - Starting task: attempt_local1388077210_0023_m_000000_0
2015-02-12 16:12:51,618 INFO util.ProcfsBasedProcessTree (ProcfsBasedProcessTree.java:isAvailable(181)) - ProcfsBasedProcessTree currently is supported only on Linux.
2015-02-12 16:12:51,618 INFO mapred.Task (Task.java:initialize(587)) - Using ResourceCalculatorProcessTree : null
2015-02-12 16:12:51,619 INFO io.MultiInputSplit (MultiInputSplit.java:readFields(161)) - current split input path: file:/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/classes/test/data/lhs.txt
2015-02-12 16:12:51,619 INFO mapred.MapTask (MapTask.java:updateJobWithSplit(482)) - Processing split: cascading.tap.hadoop.io.MultiInputSplit@17ce3e76
2015-02-12 16:12:51,620 INFO mapred.MapTask (MapTask.java:runOldMapper(436)) - numReduceTasks: 1
2015-02-12 16:12:51,636 INFO mapred.MapTask (MapTask.java:setEquator(1202)) - (EQUATOR) 0 kvi 26214396(104857584)
2015-02-12 16:12:51,636 INFO mapred.MapTask (MapTask.java:init(995)) - mapreduce.task.io.sort.mb: 100
2015-02-12 16:12:51,636 INFO mapred.MapTask (MapTask.java:init(996)) - soft limit at 83886080
2015-02-12 16:12:51,636 INFO mapred.MapTask (MapTask.java:init(997)) - bufstart = 0; bufvoid = 104857600
2015-02-12 16:12:51,636 INFO mapred.MapTask (MapTask.java:init(998)) - kvstart = 26214396; length = 6553600
2015-02-12 16:12:51,637 INFO mapred.MapTask (MapTask.java:createSortingCollector(402)) - Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer
2015-02-12 16:12:51,637 INFO hadoop.FlowMapper (FlowMapper.java:configure(71)) - cascading version: 2.6.3
2015-02-12 16:12:51,637 INFO hadoop.FlowMapper (FlowMapper.java:configure(72)) - child jvm opts: -Xmx200m
2015-02-12 16:12:51,640 INFO hadoop.FlowMapper (FlowMapper.java:configure(87)) - sourcing from: Hfs["TextDelimited[['num', 'char']]"]["/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/classes/test/data/lhs.txt"]
2015-02-12 16:12:51,640 INFO hadoop.FlowMapper (FlowMapper.java:configure(90)) - sinking to: GroupBy(max)[by:[{1}:'char']]
2015-02-12 16:12:51,641 INFO assembly.AggregateBy (AggregateBy.java:prepare(267)) - using threshold value: 2
2015-02-12 16:12:51,641 INFO assembly.AggregateBy (AggregateBy.java:removeEldestEntry(327)) - flushed keys num times: 1, with threshold: 2
2015-02-12 16:12:51,641 INFO assembly.AggregateBy (AggregateBy.java:removeEldestEntry(328)) - mem on flush (mb), free: 196, total: 511, max: 672
2015-02-12 16:12:51,642 INFO assembly.AggregateBy (AggregateBy.java:removeEldestEntry(333)) - total mem is 76% of max mem, to better utilize unused memory consider increasing current LRU threshold with system property "cascading.aggregateby.threshold"
2015-02-12 16:12:51,642 INFO assembly.AggregateBy (AggregateBy.java:removeEldestEntry(327)) - flushed keys num times: 3, with threshold: 2
2015-02-12 16:12:51,642 INFO assembly.AggregateBy (AggregateBy.java:removeEldestEntry(328)) - mem on flush (mb), free: 196, total: 511, max: 672
2015-02-12 16:12:51,642 INFO assembly.AggregateBy (AggregateBy.java:removeEldestEntry(333)) - total mem is 76% of max mem, to better utilize unused memory consider increasing current LRU threshold with system property "cascading.aggregateby.threshold"
2015-02-12 16:12:51,643 INFO assembly.AggregateBy (AggregateBy.java:removeEldestEntry(327)) - flushed keys num times: 5, with threshold: 2
2015-02-12 16:12:51,643 INFO assembly.AggregateBy (AggregateBy.java:removeEldestEntry(328)) - mem on flush (mb), free: 196, total: 511, max: 672
2015-02-12 16:12:51,643 INFO assembly.AggregateBy (AggregateBy.java:removeEldestEntry(333)) - total mem is 76% of max mem, to better utilize unused memory consider increasing current LRU threshold with system property "cascading.aggregateby.threshold"
2015-02-12 16:12:51,643 INFO assembly.AggregateBy (AggregateBy.java:removeEldestEntry(327)) - flushed keys num times: 7, with threshold: 2
2015-02-12 16:12:51,643 INFO assembly.AggregateBy (AggregateBy.java:removeEldestEntry(328)) - mem on flush (mb), free: 196, total: 511, max: 672
2015-02-12 16:12:51,643 INFO assembly.AggregateBy (AggregateBy.java:removeEldestEntry(333)) - total mem is 76% of max mem, to better utilize unused memory consider increasing current LRU threshold with system property "cascading.aggregateby.threshold"
2015-02-12 16:12:51,644 INFO mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) -
2015-02-12 16:12:51,644 INFO mapred.MapTask (MapTask.java:flush(1457)) - Starting flush of map output
2015-02-12 16:12:51,644 INFO mapred.MapTask (MapTask.java:flush(1475)) - Spilling map output
2015-02-12 16:12:51,644 INFO mapred.MapTask (MapTask.java:flush(1476)) - bufstart = 0; bufend = 98; bufvoid = 104857600
2015-02-12 16:12:51,644 INFO mapred.MapTask (MapTask.java:flush(1478)) - kvstart = 26214396(104857584); kvend = 26214364(104857456); length = 33/6553600
2015-02-12 16:12:51,646 INFO mapred.MapTask (MapTask.java:sortAndSpill(1660)) - Finished spill 0
2015-02-12 16:12:51,647 INFO mapred.Task (Task.java:done(1001)) - Task:attempt_local1388077210_0023_m_000000_0 is done. And is in the process of committing
2015-02-12 16:12:51,648 INFO mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) - file:/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/classes/test/data/lhs.txt:0+52
2015-02-12 16:12:51,648 INFO mapred.Task (Task.java:sendDone(1121)) - Task 'attempt_local1388077210_0023_m_000000_0' done.
2015-02-12 16:12:51,648 INFO mapred.LocalJobRunner (LocalJobRunner.java:run(249)) - Finishing task: attempt_local1388077210_0023_m_000000_0
2015-02-12 16:12:51,648 INFO mapred.LocalJobRunner (LocalJobRunner.java:runTasks(456)) - map task executor complete.
2015-02-12 16:12:51,649 INFO mapred.LocalJobRunner (LocalJobRunner.java:runTasks(448)) - Waiting for reduce tasks
2015-02-12 16:12:51,649 INFO mapred.LocalJobRunner (LocalJobRunner.java:run(302)) - Starting task: attempt_local1388077210_0023_r_000000_0
2015-02-12 16:12:51,650 INFO util.ProcfsBasedProcessTree (ProcfsBasedProcessTree.java:isAvailable(181)) - ProcfsBasedProcessTree currently is supported only on Linux.
2015-02-12 16:12:51,650 INFO mapred.Task (Task.java:initialize(587)) - Using ResourceCalculatorProcessTree : null
2015-02-12 16:12:51,650 INFO mapred.ReduceTask (ReduceTask.java:run(362)) - Using ShuffleConsumerPlugin: org.apache.hadoop.mapreduce.task.reduce.Shuffle@4f817cc9
2015-02-12 16:12:51,650 INFO reduce.MergeManagerImpl (MergeManagerImpl.java:<init>(195)) - MergerManager: memoryLimit=493250144, maxSingleShuffleLimit=123312536, mergeThreshold=325545120, ioSortFactor=10, memToMemMergeOutputsThreshold=10
2015-02-12 16:12:51,651 INFO reduce.EventFetcher (EventFetcher.java:run(61)) - attempt_local1388077210_0023_r_000000_0 Thread started: EventFetcher for fetching Map Completion Events
2015-02-12 16:12:51,652 INFO reduce.LocalFetcher (LocalFetcher.java:copyMapOutput(141)) - localfetcher#19 about to shuffle output of map attempt_local1388077210_0023_m_000000_0 decomp: 118 len: 122 to MEMORY
2015-02-12 16:12:51,652 INFO reduce.InMemoryMapOutput (InMemoryMapOutput.java:shuffle(100)) - Read 118 bytes from map-output for attempt_local1388077210_0023_m_000000_0
2015-02-12 16:12:51,652 INFO reduce.MergeManagerImpl (MergeManagerImpl.java:closeInMemoryFile(313)) - closeInMemoryFile -> map-output of size: 118, inMemoryMapOutputs.size() -> 1, commitMemory -> 0, usedMemory ->118
2015-02-12 16:12:51,652 INFO reduce.EventFetcher (EventFetcher.java:run(76)) - EventFetcher is interrupted.. Returning
2015-02-12 16:12:51,653 INFO mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) - 1 / 1 copied.
2015-02-12 16:12:51,653 INFO reduce.MergeManagerImpl (MergeManagerImpl.java:finalMerge(673)) - finalMerge called with 1 in-memory map-outputs and 0 on-disk map-outputs
2015-02-12 16:12:51,654 INFO mapred.Merger (Merger.java:merge(606)) - Merging 1 sorted segments
2015-02-12 16:12:51,654 INFO mapred.Merger (Merger.java:merge(705)) - Down to the last merge-pass, with 1 segments left of total size: 109 bytes
2015-02-12 16:12:51,655 INFO reduce.MergeManagerImpl (MergeManagerImpl.java:finalMerge(750)) - Merged 1 segments, 118 bytes to disk to satisfy reduce memory limit
2015-02-12 16:12:51,655 INFO reduce.MergeManagerImpl (MergeManagerImpl.java:finalMerge(780)) - Merging 1 files, 122 bytes from disk
2015-02-12 16:12:51,655 INFO reduce.MergeManagerImpl (MergeManagerImpl.java:finalMerge(795)) - Merging 0 segments, 0 bytes from memory into reduce
2015-02-12 16:12:51,656 INFO mapred.Merger (Merger.java:merge(606)) - Merging 1 sorted segments
2015-02-12 16:12:51,656 INFO mapred.Merger (Merger.java:merge(705)) - Down to the last merge-pass, with 1 segments left of total size: 109 bytes
2015-02-12 16:12:51,656 INFO mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) - 1 / 1 copied.
2015-02-12 16:12:51,657 INFO hadoop.FlowReducer (FlowReducer.java:configure(78)) - cascading version: 2.6.3
2015-02-12 16:12:51,657 INFO hadoop.FlowReducer (FlowReducer.java:configure(79)) - child jvm opts: -Xmx200m
2015-02-12 16:12:51,661 INFO hadoop.FlowReducer (FlowReducer.java:configure(97)) - sourcing from: GroupBy(max)[by:[{1}:'char']]
2015-02-12 16:12:51,661 INFO hadoop.FlowReducer (FlowReducer.java:configure(100)) - sinking to: Hfs["TextDelimited[['char', 'max']]"]["/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/test/output/hadoop2-mr1/assemblyhelpersplatform/maxbynullsafety"]
2015-02-12 16:12:51,668 INFO mapred.Task (Task.java:done(1001)) - Task:attempt_local1388077210_0023_r_000000_0 is done. And is in the process of committing
2015-02-12 16:12:51,668 INFO mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) - 1 / 1 copied.
2015-02-12 16:12:51,669 INFO mapred.Task (Task.java:commit(1162)) - Task attempt_local1388077210_0023_r_000000_0 is allowed to commit now
2015-02-12 16:12:51,669 INFO output.FileOutputCommitter (FileOutputCommitter.java:commitTask(439)) - Saved output of task 'attempt_local1388077210_0023_r_000000_0' to file:/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/test/output/hadoop2-mr1/assemblyhelpersplatform/maxbynullsafety/_temporary/0/task_local1388077210_0023_r_000000
2015-02-12 16:12:51,670 INFO mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) - reduce > reduce
2015-02-12 16:12:51,670 INFO mapred.Task (Task.java:sendDone(1121)) - Task 'attempt_local1388077210_0023_r_000000_0' done.
2015-02-12 16:12:51,670 INFO mapred.LocalJobRunner (LocalJobRunner.java:run(325)) - Finishing task: attempt_local1388077210_0023_r_000000_0
2015-02-12 16:12:51,670 INFO mapred.LocalJobRunner (LocalJobRunner.java:runTasks(456)) - reduce task executor complete.
2015-02-12 16:12:51,688 INFO util.Hadoop18TapUtil (Hadoop18TapUtil.java:cleanTempPath(219)) - deleting temp path /Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/test/output/hadoop2-mr1/assemblyhelpersplatform/maxbynullsafety/_temporary
2015-02-12 16:12:51,708 INFO mapred.FileInputFormat (FileInputFormat.java:listStatus(247)) - Total input paths to process : 1
2015-02-12 16:12:51,732 INFO mapred.FileInputFormat (FileInputFormat.java:listStatus(247)) - Total input paths to process : 1
2015-02-12 16:12:51,733 INFO cascading.PlatformTestCase (PlatformTestCase.java:tearDown(209)) - copying to local /Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/test/output//hadoop2-mr1/assemblyhelpersplatform/maxbynullsafety
2015-02-12 16:12:51,744 INFO util.HadoopUtil (HadoopUtil.java:findMainClass(336)) - using default application jar, may cause class not found exceptions on the cluster
2015-02-12 16:12:51,744 INFO planner.HadoopPlanner (HadoopPlanner.java:initialize(225)) - using application jar: /Users/soberoi/.gradle/caches/modules-2/files-2.1/cascading/cascading-hadoop2-mr1/2.6.3/9ae016e2b8d246b9c1de9168e715d9f1b971aa4a/cascading-hadoop2-mr1-2.6.3.jar
2015-02-12 16:12:51,748 INFO Configuration.deprecation (Configuration.java:warnOnceIfDeprecated(1022)) - mapred.used.genericoptionsparser is deprecated. Instead, use mapreduce.client.genericoptionsparser.used
2015-02-12 16:12:51,749 INFO Configuration.deprecation (Configuration.java:warnOnceIfDeprecated(1022)) - mapred.job.tracker is deprecated. Instead, use mapreduce.jobtracker.address
2015-02-12 16:12:51,749 INFO Configuration.deprecation (Configuration.java:warnOnceIfDeprecated(1022)) - mapred.output.compress is deprecated. Instead, use mapreduce.output.fileoutputformat.compress
2015-02-12 16:12:51,754 INFO flow.Flow (BaseFlow.java:logInfo(1378)) - [sum] starting
2015-02-12 16:12:51,754 INFO flow.Flow (BaseFlow.java:logInfo(1378)) - [sum] source: Hfs["TextDelimited[['num', 'char']]"]["/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/classes/test/data/lhs.txt"]
2015-02-12 16:12:51,754 INFO flow.Flow (BaseFlow.java:logInfo(1378)) - [sum] sink: Hfs["TextDelimited[['char', 'sum']]"]["/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/test/output/hadoop2-mr1/assemblyhelpersplatform/sumnulls"]
2015-02-12 16:12:51,754 INFO flow.Flow (BaseFlow.java:logInfo(1378)) - [sum] parallel execution is enabled: false
2015-02-12 16:12:51,754 INFO flow.Flow (BaseFlow.java:logInfo(1378)) - [sum] starting jobs: 1
2015-02-12 16:12:51,754 INFO flow.Flow (BaseFlow.java:logInfo(1378)) - [sum] allocating threads: 1
2015-02-12 16:12:51,755 INFO flow.FlowStep (BaseFlowStep.java:logInfo(834)) - [sum] starting step: (1/1) ...yhelpersplatform/sumnulls
2015-02-12 16:12:51,756 INFO jvm.JvmMetrics (JvmMetrics.java:init(71)) - Cannot initialize JVM Metrics with processName=JobTracker, sessionId= - already initialized
2015-02-12 16:12:51,756 INFO jvm.JvmMetrics (JvmMetrics.java:init(71)) - Cannot initialize JVM Metrics with processName=JobTracker, sessionId= - already initialized
2015-02-12 16:12:51,775 INFO mapred.FileInputFormat (FileInputFormat.java:listStatus(247)) - Total input paths to process : 1
2015-02-12 16:12:51,791 INFO mapreduce.JobSubmitter (JobSubmitter.java:submitJobInternal(492)) - number of splits:1
2015-02-12 16:12:51,802 INFO mapreduce.JobSubmitter (JobSubmitter.java:printTokens(575)) - Submitting tokens for job: job_local1795518538_0024
2015-02-12 16:12:51,813 WARN conf.Configuration (Configuration.java:loadProperty(2510)) - file:/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/tmp/cascading/staging/soberoi1795518538/.staging/job_local1795518538_0024/job.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.retry.interval; Ignoring.
2015-02-12 16:12:51,814 WARN conf.Configuration (Configuration.java:loadProperty(2510)) - file:/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/tmp/cascading/staging/soberoi1795518538/.staging/job_local1795518538_0024/job.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.attempts; Ignoring.
2015-02-12 16:12:51,846 WARN conf.Configuration (Configuration.java:loadProperty(2510)) - file:/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/test/tmp/mapred/local/localRunner/soberoi/job_local1795518538_0024/job_local1795518538_0024.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.retry.interval; Ignoring.
2015-02-12 16:12:51,847 WARN conf.Configuration (Configuration.java:loadProperty(2510)) - file:/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/test/tmp/mapred/local/localRunner/soberoi/job_local1795518538_0024/job_local1795518538_0024.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.attempts; Ignoring.
2015-02-12 16:12:51,847 INFO mapreduce.Job (Job.java:submit(1299)) - The url to track the job: http://localhost:8080/
2015-02-12 16:12:51,847 INFO mapred.LocalJobRunner (LocalJobRunner.java:createOutputCommitter(471)) - OutputCommitter set in config null
2015-02-12 16:12:51,847 INFO flow.FlowStep (BaseFlowStep.java:logInfo(834)) - [sum] submitted hadoop job: job_local1795518538_0024
2015-02-12 16:12:51,847 INFO mapred.LocalJobRunner (LocalJobRunner.java:createOutputCommitter(489)) - OutputCommitter is org.apache.hadoop.mapred.FileOutputCommitter
2015-02-12 16:12:51,847 INFO flow.FlowStep (BaseFlowStep.java:logInfo(834)) - [sum] tracking url: http://localhost:8080/
2015-02-12 16:12:51,849 INFO mapred.LocalJobRunner (LocalJobRunner.java:runTasks(448)) - Waiting for map tasks
2015-02-12 16:12:51,849 INFO mapred.LocalJobRunner (LocalJobRunner.java:run(224)) - Starting task: attempt_local1795518538_0024_m_000000_0
2015-02-12 16:12:51,850 INFO util.ProcfsBasedProcessTree (ProcfsBasedProcessTree.java:isAvailable(181)) - ProcfsBasedProcessTree currently is supported only on Linux.
2015-02-12 16:12:51,850 INFO mapred.Task (Task.java:initialize(587)) - Using ResourceCalculatorProcessTree : null
2015-02-12 16:12:51,851 INFO io.MultiInputSplit (MultiInputSplit.java:readFields(161)) - current split input path: file:/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/classes/test/data/lhs.txt
2015-02-12 16:12:51,851 INFO mapred.MapTask (MapTask.java:updateJobWithSplit(482)) - Processing split: cascading.tap.hadoop.io.MultiInputSplit@378577ad
2015-02-12 16:12:51,851 INFO mapred.MapTask (MapTask.java:runOldMapper(436)) - numReduceTasks: 1
2015-02-12 16:12:51,867 INFO mapred.MapTask (MapTask.java:setEquator(1202)) - (EQUATOR) 0 kvi 26214396(104857584)
2015-02-12 16:12:51,867 INFO mapred.MapTask (MapTask.java:init(995)) - mapreduce.task.io.sort.mb: 100
2015-02-12 16:12:51,867 INFO mapred.MapTask (MapTask.java:init(996)) - soft limit at 83886080
2015-02-12 16:12:51,867 INFO mapred.MapTask (MapTask.java:init(997)) - bufstart = 0; bufvoid = 104857600
2015-02-12 16:12:51,867 INFO mapred.MapTask (MapTask.java:init(998)) - kvstart = 26214396; length = 6553600
2015-02-12 16:12:51,868 INFO mapred.MapTask (MapTask.java:createSortingCollector(402)) - Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer
2015-02-12 16:12:51,868 INFO hadoop.FlowMapper (FlowMapper.java:configure(71)) - cascading version: 2.6.3
2015-02-12 16:12:51,868 INFO hadoop.FlowMapper (FlowMapper.java:configure(72)) - child jvm opts: -Xmx200m
2015-02-12 16:12:51,871 INFO hadoop.FlowMapper (FlowMapper.java:configure(87)) - sourcing from: Hfs["TextDelimited[['num', 'char']]"]["/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/classes/test/data/lhs.txt"]
2015-02-12 16:12:51,871 INFO hadoop.FlowMapper (FlowMapper.java:configure(90)) - sinking to: GroupBy(sum)[by:[{1}:'char']]
2015-02-12 16:12:51,872 INFO assembly.AggregateBy (AggregateBy.java:prepare(267)) - using threshold value: 2
2015-02-12 16:12:51,876 INFO assembly.AggregateBy (AggregateBy.java:removeEldestEntry(327)) - flushed keys num times: 1, with threshold: 2
2015-02-12 16:12:51,876 INFO assembly.AggregateBy (AggregateBy.java:removeEldestEntry(328)) - mem on flush (mb), free: 196, total: 512, max: 672
2015-02-12 16:12:51,876 INFO assembly.AggregateBy (AggregateBy.java:removeEldestEntry(333)) - total mem is 76% of max mem, to better utilize unused memory consider increasing current LRU threshold with system property "cascading.aggregateby.threshold"
2015-02-12 16:12:51,877 INFO assembly.AggregateBy (AggregateBy.java:removeEldestEntry(327)) - flushed keys num times: 3, with threshold: 2
2015-02-12 16:12:51,877 INFO assembly.AggregateBy (AggregateBy.java:removeEldestEntry(328)) - mem on flush (mb), free: 196, total: 512, max: 672
2015-02-12 16:12:51,877 INFO assembly.AggregateBy (AggregateBy.java:removeEldestEntry(333)) - total mem is 76% of max mem, to better utilize unused memory consider increasing current LRU threshold with system property "cascading.aggregateby.threshold"
2015-02-12 16:12:51,878 INFO assembly.AggregateBy (AggregateBy.java:removeEldestEntry(327)) - flushed keys num times: 5, with threshold: 2
2015-02-12 16:12:51,878 INFO assembly.AggregateBy (AggregateBy.java:removeEldestEntry(328)) - mem on flush (mb), free: 196, total: 512, max: 672
2015-02-12 16:12:51,878 INFO assembly.AggregateBy (AggregateBy.java:removeEldestEntry(333)) - total mem is 76% of max mem, to better utilize unused memory consider increasing current LRU threshold with system property "cascading.aggregateby.threshold"
2015-02-12 16:12:51,878 INFO assembly.AggregateBy (AggregateBy.java:removeEldestEntry(327)) - flushed keys num times: 7, with threshold: 2
2015-02-12 16:12:51,878 INFO assembly.AggregateBy (AggregateBy.java:removeEldestEntry(328)) - mem on flush (mb), free: 196, total: 512, max: 672
2015-02-12 16:12:51,878 INFO assembly.AggregateBy (AggregateBy.java:removeEldestEntry(333)) - total mem is 76% of max mem, to better utilize unused memory consider increasing current LRU threshold with system property "cascading.aggregateby.threshold"
2015-02-12 16:12:51,879 INFO mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) -
2015-02-12 16:12:51,879 INFO mapred.MapTask (MapTask.java:flush(1457)) - Starting flush of map output
2015-02-12 16:12:51,879 INFO mapred.MapTask (MapTask.java:flush(1475)) - Spilling map output
2015-02-12 16:12:51,880 INFO mapred.MapTask (MapTask.java:flush(1476)) - bufstart = 0; bufend = 96; bufvoid = 104857600
2015-02-12 16:12:51,880 INFO mapred.MapTask (MapTask.java:flush(1478)) - kvstart = 26214396(104857584); kvend = 26214364(104857456); length = 33/6553600
2015-02-12 16:12:51,881 INFO mapred.MapTask (MapTask.java:sortAndSpill(1660)) - Finished spill 0
2015-02-12 16:12:51,883 INFO mapred.Task (Task.java:done(1001)) - Task:attempt_local1795518538_0024_m_000000_0 is done. And is in the process of committing
2015-02-12 16:12:51,884 INFO mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) - file:/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/classes/test/data/lhs.txt:0+52
2015-02-12 16:12:51,884 INFO mapred.Task (Task.java:sendDone(1121)) - Task 'attempt_local1795518538_0024_m_000000_0' done.
2015-02-12 16:12:51,884 INFO mapred.LocalJobRunner (LocalJobRunner.java:run(249)) - Finishing task: attempt_local1795518538_0024_m_000000_0
2015-02-12 16:12:51,884 INFO mapred.LocalJobRunner (LocalJobRunner.java:runTasks(456)) - map task executor complete.
2015-02-12 16:12:51,884 INFO mapred.LocalJobRunner (LocalJobRunner.java:runTasks(448)) - Waiting for reduce tasks
2015-02-12 16:12:51,884 INFO mapred.LocalJobRunner (LocalJobRunner.java:run(302)) - Starting task: attempt_local1795518538_0024_r_000000_0
2015-02-12 16:12:51,885 INFO util.ProcfsBasedProcessTree (ProcfsBasedProcessTree.java:isAvailable(181)) - ProcfsBasedProcessTree currently is supported only on Linux.
2015-02-12 16:12:51,886 INFO mapred.Task (Task.java:initialize(587)) - Using ResourceCalculatorProcessTree : null
2015-02-12 16:12:51,886 INFO mapred.ReduceTask (ReduceTask.java:run(362)) - Using ShuffleConsumerPlugin: org.apache.hadoop.mapreduce.task.reduce.Shuffle@7ac7e297
2015-02-12 16:12:51,886 INFO reduce.MergeManagerImpl (MergeManagerImpl.java:<init>(195)) - MergerManager: memoryLimit=493250144, maxSingleShuffleLimit=123312536, mergeThreshold=325545120, ioSortFactor=10, memToMemMergeOutputsThreshold=10
2015-02-12 16:12:51,886 INFO reduce.EventFetcher (EventFetcher.java:run(61)) - attempt_local1795518538_0024_r_000000_0 Thread started: EventFetcher for fetching Map Completion Events
2015-02-12 16:12:51,887 INFO reduce.LocalFetcher (LocalFetcher.java:copyMapOutput(141)) - localfetcher#20 about to shuffle output of map attempt_local1795518538_0024_m_000000_0 decomp: 116 len: 120 to MEMORY
2015-02-12 16:12:51,888 INFO reduce.InMemoryMapOutput (InMemoryMapOutput.java:shuffle(100)) - Read 116 bytes from map-output for attempt_local1795518538_0024_m_000000_0
2015-02-12 16:12:51,888 INFO reduce.MergeManagerImpl (MergeManagerImpl.java:closeInMemoryFile(313)) - closeInMemoryFile -> map-output of size: 116, inMemoryMapOutputs.size() -> 1, commitMemory -> 0, usedMemory ->116
2015-02-12 16:12:51,888 INFO reduce.EventFetcher (EventFetcher.java:run(76)) - EventFetcher is interrupted.. Returning
2015-02-12 16:12:51,889 INFO mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) - 1 / 1 copied.
2015-02-12 16:12:51,889 INFO reduce.MergeManagerImpl (MergeManagerImpl.java:finalMerge(673)) - finalMerge called with 1 in-memory map-outputs and 0 on-disk map-outputs
2015-02-12 16:12:51,890 INFO mapred.Merger (Merger.java:merge(606)) - Merging 1 sorted segments
2015-02-12 16:12:51,890 INFO mapred.Merger (Merger.java:merge(705)) - Down to the last merge-pass, with 1 segments left of total size: 107 bytes
2015-02-12 16:12:51,891 INFO reduce.MergeManagerImpl (MergeManagerImpl.java:finalMerge(750)) - Merged 1 segments, 116 bytes to disk to satisfy reduce memory limit
2015-02-12 16:12:51,891 INFO reduce.MergeManagerImpl (MergeManagerImpl.java:finalMerge(780)) - Merging 1 files, 120 bytes from disk
2015-02-12 16:12:51,891 INFO reduce.MergeManagerImpl (MergeManagerImpl.java:finalMerge(795)) - Merging 0 segments, 0 bytes from memory into reduce
2015-02-12 16:12:51,892 INFO mapred.Merger (Merger.java:merge(606)) - Merging 1 sorted segments
2015-02-12 16:12:51,892 INFO mapred.Merger (Merger.java:merge(705)) - Down to the last merge-pass, with 1 segments left of total size: 107 bytes
2015-02-12 16:12:51,892 INFO mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) - 1 / 1 copied.
2015-02-12 16:12:51,893 INFO hadoop.FlowReducer (FlowReducer.java:configure(78)) - cascading version: 2.6.3
2015-02-12 16:12:51,893 INFO hadoop.FlowReducer (FlowReducer.java:configure(79)) - child jvm opts: -Xmx200m
2015-02-12 16:12:51,897 INFO hadoop.FlowReducer (FlowReducer.java:configure(97)) - sourcing from: GroupBy(sum)[by:[{1}:'char']]
2015-02-12 16:12:51,897 INFO hadoop.FlowReducer (FlowReducer.java:configure(100)) - sinking to: Hfs["TextDelimited[['char', 'sum']]"]["/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/test/output/hadoop2-mr1/assemblyhelpersplatform/sumnulls"]
2015-02-12 16:12:51,903 INFO mapred.Task (Task.java:done(1001)) - Task:attempt_local1795518538_0024_r_000000_0 is done. And is in the process of committing
2015-02-12 16:12:51,904 INFO mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) - 1 / 1 copied.
2015-02-12 16:12:51,904 INFO mapred.Task (Task.java:commit(1162)) - Task attempt_local1795518538_0024_r_000000_0 is allowed to commit now
2015-02-12 16:12:51,905 INFO output.FileOutputCommitter (FileOutputCommitter.java:commitTask(439)) - Saved output of task 'attempt_local1795518538_0024_r_000000_0' to file:/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/test/output/hadoop2-mr1/assemblyhelpersplatform/sumnulls/_temporary/0/task_local1795518538_0024_r_000000
2015-02-12 16:12:51,905 INFO mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) - reduce > reduce
2015-02-12 16:12:51,905 INFO mapred.Task (Task.java:sendDone(1121)) - Task 'attempt_local1795518538_0024_r_000000_0' done.
2015-02-12 16:12:51,905 INFO mapred.LocalJobRunner (LocalJobRunner.java:run(325)) - Finishing task: attempt_local1795518538_0024_r_000000_0
2015-02-12 16:12:51,905 INFO mapred.LocalJobRunner (LocalJobRunner.java:runTasks(456)) - reduce task executor complete.
2015-02-12 16:12:51,927 INFO util.Hadoop18TapUtil (Hadoop18TapUtil.java:cleanTempPath(219)) - deleting temp path /Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/test/output/hadoop2-mr1/assemblyhelpersplatform/sumnulls/_temporary
2015-02-12 16:12:51,946 INFO mapred.FileInputFormat (FileInputFormat.java:listStatus(247)) - Total input paths to process : 1
2015-02-12 16:12:51,966 INFO mapred.FileInputFormat (FileInputFormat.java:listStatus(247)) - Total input paths to process : 1
2015-02-12 16:12:51,967 INFO cascading.PlatformTestCase (PlatformTestCase.java:tearDown(209)) - copying to local /Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/test/output//hadoop2-mr1/assemblyhelpersplatform/sumnulls
2015-02-12 16:12:51,981 INFO util.HadoopUtil (HadoopUtil.java:findMainClass(336)) - using default application jar, may cause class not found exceptions on the cluster
2015-02-12 16:12:51,981 INFO planner.HadoopPlanner (HadoopPlanner.java:initialize(225)) - using application jar: /Users/soberoi/.gradle/caches/modules-2/files-2.1/cascading/cascading-hadoop2-mr1/2.6.3/9ae016e2b8d246b9c1de9168e715d9f1b971aa4a/cascading-hadoop2-mr1-2.6.3.jar
2015-02-12 16:12:51,984 INFO Configuration.deprecation (Configuration.java:warnOnceIfDeprecated(1022)) - mapred.used.genericoptionsparser is deprecated. Instead, use mapreduce.client.genericoptionsparser.used
2015-02-12 16:12:51,985 INFO Configuration.deprecation (Configuration.java:warnOnceIfDeprecated(1022)) - mapred.job.tracker is deprecated. Instead, use mapreduce.jobtracker.address
2015-02-12 16:12:51,985 INFO Configuration.deprecation (Configuration.java:warnOnceIfDeprecated(1022)) - mapred.output.compress is deprecated. Instead, use mapreduce.output.fileoutputformat.compress
2015-02-12 16:12:51,989 INFO flow.Flow (BaseFlow.java:logInfo(1378)) - [count] starting
2015-02-12 16:12:51,990 INFO flow.Flow (BaseFlow.java:logInfo(1378)) - [count] source: Hfs["TextDelimited[['num', 'char']]"]["/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/classes/test/data/lhs.txt"]
2015-02-12 16:12:51,990 INFO flow.Flow (BaseFlow.java:logInfo(1378)) - [count] sink: Hfs["TextDelimited[['count']]"]["/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/test/output/hadoop2-mr1/assemblyhelpersplatform/countall"]
2015-02-12 16:12:51,990 INFO flow.Flow (BaseFlow.java:logInfo(1378)) - [count] parallel execution is enabled: false
2015-02-12 16:12:51,990 INFO flow.Flow (BaseFlow.java:logInfo(1378)) - [count] starting jobs: 1
2015-02-12 16:12:51,990 INFO flow.Flow (BaseFlow.java:logInfo(1378)) - [count] allocating threads: 1
2015-02-12 16:12:51,990 INFO flow.FlowStep (BaseFlowStep.java:logInfo(834)) - [count] starting step: (1/1) ...yhelpersplatform/countall
2015-02-12 16:12:51,991 INFO jvm.JvmMetrics (JvmMetrics.java:init(71)) - Cannot initialize JVM Metrics with processName=JobTracker, sessionId= - already initialized
2015-02-12 16:12:51,991 INFO jvm.JvmMetrics (JvmMetrics.java:init(71)) - Cannot initialize JVM Metrics with processName=JobTracker, sessionId= - already initialized
2015-02-12 16:12:52,010 INFO mapred.FileInputFormat (FileInputFormat.java:listStatus(247)) - Total input paths to process : 1
2015-02-12 16:12:52,025 INFO mapreduce.JobSubmitter (JobSubmitter.java:submitJobInternal(492)) - number of splits:1
2015-02-12 16:12:52,037 INFO mapreduce.JobSubmitter (JobSubmitter.java:printTokens(575)) - Submitting tokens for job: job_local375512920_0025
2015-02-12 16:12:52,048 WARN conf.Configuration (Configuration.java:loadProperty(2510)) - file:/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/tmp/cascading/staging/soberoi375512920/.staging/job_local375512920_0025/job.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.retry.interval; Ignoring.
2015-02-12 16:12:52,049 WARN conf.Configuration (Configuration.java:loadProperty(2510)) - file:/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/tmp/cascading/staging/soberoi375512920/.staging/job_local375512920_0025/job.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.attempts; Ignoring.
2015-02-12 16:12:52,078 WARN conf.Configuration (Configuration.java:loadProperty(2510)) - file:/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/test/tmp/mapred/local/localRunner/soberoi/job_local375512920_0025/job_local375512920_0025.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.retry.interval; Ignoring.
2015-02-12 16:12:52,079 WARN conf.Configuration (Configuration.java:loadProperty(2510)) - file:/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/test/tmp/mapred/local/localRunner/soberoi/job_local375512920_0025/job_local375512920_0025.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.attempts; Ignoring.
2015-02-12 16:12:52,080 INFO mapreduce.Job (Job.java:submit(1299)) - The url to track the job: http://localhost:8080/
2015-02-12 16:12:52,080 INFO flow.FlowStep (BaseFlowStep.java:logInfo(834)) - [count] submitted hadoop job: job_local375512920_0025
2015-02-12 16:12:52,080 INFO mapred.LocalJobRunner (LocalJobRunner.java:createOutputCommitter(471)) - OutputCommitter set in config null
2015-02-12 16:12:52,080 INFO flow.FlowStep (BaseFlowStep.java:logInfo(834)) - [count] tracking url: http://localhost:8080/
2015-02-12 16:12:52,081 INFO mapred.LocalJobRunner (LocalJobRunner.java:createOutputCommitter(489)) - OutputCommitter is org.apache.hadoop.mapred.FileOutputCommitter
2015-02-12 16:12:52,082 INFO mapred.LocalJobRunner (LocalJobRunner.java:runTasks(448)) - Waiting for map tasks
2015-02-12 16:12:52,082 INFO mapred.LocalJobRunner (LocalJobRunner.java:run(224)) - Starting task: attempt_local375512920_0025_m_000000_0
2015-02-12 16:12:52,083 INFO util.ProcfsBasedProcessTree (ProcfsBasedProcessTree.java:isAvailable(181)) - ProcfsBasedProcessTree currently is supported only on Linux.
2015-02-12 16:12:52,083 INFO mapred.Task (Task.java:initialize(587)) - Using ResourceCalculatorProcessTree : null
2015-02-12 16:12:52,084 INFO io.MultiInputSplit (MultiInputSplit.java:readFields(161)) - current split input path: file:/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/classes/test/data/lhs.txt
2015-02-12 16:12:52,084 INFO mapred.MapTask (MapTask.java:updateJobWithSplit(482)) - Processing split: cascading.tap.hadoop.io.MultiInputSplit@7433ccf0
2015-02-12 16:12:52,085 INFO mapred.MapTask (MapTask.java:runOldMapper(436)) - numReduceTasks: 1
2015-02-12 16:12:52,101 INFO mapred.MapTask (MapTask.java:setEquator(1202)) - (EQUATOR) 0 kvi 26214396(104857584)
2015-02-12 16:12:52,101 INFO mapred.MapTask (MapTask.java:init(995)) - mapreduce.task.io.sort.mb: 100
2015-02-12 16:12:52,101 INFO mapred.MapTask (MapTask.java:init(996)) - soft limit at 83886080
2015-02-12 16:12:52,101 INFO mapred.MapTask (MapTask.java:init(997)) - bufstart = 0; bufvoid = 104857600
2015-02-12 16:12:52,101 INFO mapred.MapTask (MapTask.java:init(998)) - kvstart = 26214396; length = 6553600
2015-02-12 16:12:52,102 INFO mapred.MapTask (MapTask.java:createSortingCollector(402)) - Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer
2015-02-12 16:12:52,102 INFO hadoop.FlowMapper (FlowMapper.java:configure(71)) - cascading version: 2.6.3
2015-02-12 16:12:52,102 INFO hadoop.FlowMapper (FlowMapper.java:configure(72)) - child jvm opts: -Xmx200m
2015-02-12 16:12:52,106 INFO hadoop.FlowMapper (FlowMapper.java:configure(87)) - sourcing from: Hfs["TextDelimited[['num', 'char']]"]["/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/classes/test/data/lhs.txt"]
2015-02-12 16:12:52,106 INFO hadoop.FlowMapper (FlowMapper.java:configure(90)) - sinking to: GroupBy(count)[by:[{?}:NONE]]
2015-02-12 16:12:52,107 INFO assembly.AggregateBy (AggregateBy.java:prepare(267)) - using threshold value: 2
2015-02-12 16:12:52,109 INFO mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) -
2015-02-12 16:12:52,109 INFO mapred.MapTask (MapTask.java:flush(1457)) - Starting flush of map output
2015-02-12 16:12:52,109 INFO mapred.MapTask (MapTask.java:flush(1475)) - Spilling map output
2015-02-12 16:12:52,109 INFO mapred.MapTask (MapTask.java:flush(1476)) - bufstart = 0; bufend = 4; bufvoid = 104857600
2015-02-12 16:12:52,109 INFO mapred.MapTask (MapTask.java:flush(1478)) - kvstart = 26214396(104857584); kvend = 26214396(104857584); length = 1/6553600
2015-02-12 16:12:52,112 INFO mapred.MapTask (MapTask.java:sortAndSpill(1660)) - Finished spill 0
2015-02-12 16:12:52,114 INFO mapred.Task (Task.java:done(1001)) - Task:attempt_local375512920_0025_m_000000_0 is done. And is in the process of committing
2015-02-12 16:12:52,115 INFO mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) - file:/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/classes/test/data/lhs.txt:0+52
2015-02-12 16:12:52,116 INFO mapred.Task (Task.java:sendDone(1121)) - Task 'attempt_local375512920_0025_m_000000_0' done.
2015-02-12 16:12:52,116 INFO mapred.LocalJobRunner (LocalJobRunner.java:run(249)) - Finishing task: attempt_local375512920_0025_m_000000_0
2015-02-12 16:12:52,116 INFO mapred.LocalJobRunner (LocalJobRunner.java:runTasks(456)) - map task executor complete.
2015-02-12 16:12:52,116 INFO mapred.LocalJobRunner (LocalJobRunner.java:runTasks(448)) - Waiting for reduce tasks
2015-02-12 16:12:52,116 INFO mapred.LocalJobRunner (LocalJobRunner.java:run(302)) - Starting task: attempt_local375512920_0025_r_000000_0
2015-02-12 16:12:52,118 INFO util.ProcfsBasedProcessTree (ProcfsBasedProcessTree.java:isAvailable(181)) - ProcfsBasedProcessTree currently is supported only on Linux.
2015-02-12 16:12:52,118 INFO mapred.Task (Task.java:initialize(587)) - Using ResourceCalculatorProcessTree : null
2015-02-12 16:12:52,118 INFO mapred.ReduceTask (ReduceTask.java:run(362)) - Using ShuffleConsumerPlugin: org.apache.hadoop.mapreduce.task.reduce.Shuffle@42ae638
2015-02-12 16:12:52,118 INFO reduce.MergeManagerImpl (MergeManagerImpl.java:<init>(195)) - MergerManager: memoryLimit=493250144, maxSingleShuffleLimit=123312536, mergeThreshold=325545120, ioSortFactor=10, memToMemMergeOutputsThreshold=10
2015-02-12 16:12:52,119 INFO reduce.EventFetcher (EventFetcher.java:run(61)) - attempt_local375512920_0025_r_000000_0 Thread started: EventFetcher for fetching Map Completion Events
2015-02-12 16:12:52,120 INFO reduce.LocalFetcher (LocalFetcher.java:copyMapOutput(141)) - localfetcher#21 about to shuffle output of map attempt_local375512920_0025_m_000000_0 decomp: 8 len: 12 to MEMORY
2015-02-12 16:12:52,120 INFO reduce.InMemoryMapOutput (InMemoryMapOutput.java:shuffle(100)) - Read 8 bytes from map-output for attempt_local375512920_0025_m_000000_0
2015-02-12 16:12:52,120 INFO reduce.MergeManagerImpl (MergeManagerImpl.java:closeInMemoryFile(313)) - closeInMemoryFile -> map-output of size: 8, inMemoryMapOutputs.size() -> 1, commitMemory -> 0, usedMemory ->8
2015-02-12 16:12:52,121 INFO reduce.EventFetcher (EventFetcher.java:run(76)) - EventFetcher is interrupted.. Returning
2015-02-12 16:12:52,121 INFO mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) - 1 / 1 copied.
2015-02-12 16:12:52,122 INFO reduce.MergeManagerImpl (MergeManagerImpl.java:finalMerge(673)) - finalMerge called with 1 in-memory map-outputs and 0 on-disk map-outputs
2015-02-12 16:12:52,123 INFO mapred.Merger (Merger.java:merge(606)) - Merging 1 sorted segments
2015-02-12 16:12:52,123 INFO mapred.Merger (Merger.java:merge(705)) - Down to the last merge-pass, with 1 segments left of total size: 5 bytes
2015-02-12 16:12:52,124 INFO reduce.MergeManagerImpl (MergeManagerImpl.java:finalMerge(750)) - Merged 1 segments, 8 bytes to disk to satisfy reduce memory limit
2015-02-12 16:12:52,125 INFO reduce.MergeManagerImpl (MergeManagerImpl.java:finalMerge(780)) - Merging 1 files, 12 bytes from disk
2015-02-12 16:12:52,125 INFO reduce.MergeManagerImpl (MergeManagerImpl.java:finalMerge(795)) - Merging 0 segments, 0 bytes from memory into reduce
2015-02-12 16:12:52,125 INFO mapred.Merger (Merger.java:merge(606)) - Merging 1 sorted segments
2015-02-12 16:12:52,125 INFO mapred.Merger (Merger.java:merge(705)) - Down to the last merge-pass, with 1 segments left of total size: 5 bytes
2015-02-12 16:12:52,126 INFO mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) - 1 / 1 copied.
2015-02-12 16:12:52,126 INFO hadoop.FlowReducer (FlowReducer.java:configure(78)) - cascading version: 2.6.3
2015-02-12 16:12:52,126 INFO hadoop.FlowReducer (FlowReducer.java:configure(79)) - child jvm opts: -Xmx200m
2015-02-12 16:12:52,132 INFO hadoop.FlowReducer (FlowReducer.java:configure(97)) - sourcing from: GroupBy(count)[by:[{?}:NONE]]
2015-02-12 16:12:52,132 INFO hadoop.FlowReducer (FlowReducer.java:configure(100)) - sinking to: Hfs["TextDelimited[['count']]"]["/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/test/output/hadoop2-mr1/assemblyhelpersplatform/countall"]
2015-02-12 16:12:52,140 INFO mapred.Task (Task.java:done(1001)) - Task:attempt_local375512920_0025_r_000000_0 is done. And is in the process of committing
2015-02-12 16:12:52,141 INFO mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) - 1 / 1 copied.
2015-02-12 16:12:52,142 INFO mapred.Task (Task.java:commit(1162)) - Task attempt_local375512920_0025_r_000000_0 is allowed to commit now
2015-02-12 16:12:52,142 INFO output.FileOutputCommitter (FileOutputCommitter.java:commitTask(439)) - Saved output of task 'attempt_local375512920_0025_r_000000_0' to file:/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/test/output/hadoop2-mr1/assemblyhelpersplatform/countall/_temporary/0/task_local375512920_0025_r_000000
2015-02-12 16:12:52,143 INFO mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) - reduce > reduce
2015-02-12 16:12:52,143 INFO mapred.Task (Task.java:sendDone(1121)) - Task 'attempt_local375512920_0025_r_000000_0' done.
2015-02-12 16:12:52,143 INFO mapred.LocalJobRunner (LocalJobRunner.java:run(325)) - Finishing task: attempt_local375512920_0025_r_000000_0
2015-02-12 16:12:52,143 INFO mapred.LocalJobRunner (LocalJobRunner.java:runTasks(456)) - reduce task executor complete.
2015-02-12 16:12:52,164 INFO util.Hadoop18TapUtil (Hadoop18TapUtil.java:cleanTempPath(219)) - deleting temp path /Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/test/output/hadoop2-mr1/assemblyhelpersplatform/countall/_temporary
2015-02-12 16:12:52,184 INFO mapred.FileInputFormat (FileInputFormat.java:listStatus(247)) - Total input paths to process : 1
2015-02-12 16:12:52,205 INFO mapred.FileInputFormat (FileInputFormat.java:listStatus(247)) - Total input paths to process : 1
2015-02-12 16:12:52,206 INFO cascading.PlatformTestCase (PlatformTestCase.java:tearDown(209)) - copying to local /Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/test/output//hadoop2-mr1/assemblyhelpersplatform/countall
2015-02-12 16:12:52,218 INFO util.HadoopUtil (HadoopUtil.java:findMainClass(336)) - using default application jar, may cause class not found exceptions on the cluster
2015-02-12 16:12:52,219 INFO planner.HadoopPlanner (HadoopPlanner.java:initialize(225)) - using application jar: /Users/soberoi/.gradle/caches/modules-2/files-2.1/cascading/cascading-hadoop2-mr1/2.6.3/9ae016e2b8d246b9c1de9168e715d9f1b971aa4a/cascading-hadoop2-mr1-2.6.3.jar
2015-02-12 16:12:52,223 INFO Configuration.deprecation (Configuration.java:warnOnceIfDeprecated(1022)) - mapred.used.genericoptionsparser is deprecated. Instead, use mapreduce.client.genericoptionsparser.used
2015-02-12 16:12:52,224 INFO Configuration.deprecation (Configuration.java:warnOnceIfDeprecated(1022)) - mapred.job.tracker is deprecated. Instead, use mapreduce.jobtracker.address
2015-02-12 16:12:52,224 INFO Configuration.deprecation (Configuration.java:warnOnceIfDeprecated(1022)) - mapred.output.compress is deprecated. Instead, use mapreduce.output.fileoutputformat.compress
2015-02-12 16:12:52,229 INFO flow.Flow (BaseFlow.java:logInfo(1378)) - [name] starting
2015-02-12 16:12:52,229 INFO flow.Flow (BaseFlow.java:logInfo(1378)) - [name] source: Hfs["TextDelimited[['num', 'char']]"]["/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/classes/test/data/lhs.txt"]
2015-02-12 16:12:52,229 INFO flow.Flow (BaseFlow.java:logInfo(1378)) - [name] sink: Hfs["TextDelimited[['char', 'sum', 'count', 'average', 'average2', 'first']]"]["/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/test/output/hadoop2-mr1/assemblyhelpersplatform/multi"]
2015-02-12 16:12:52,229 INFO flow.Flow (BaseFlow.java:logInfo(1378)) - [name] parallel execution is enabled: false
2015-02-12 16:12:52,229 INFO flow.Flow (BaseFlow.java:logInfo(1378)) - [name] starting jobs: 1
2015-02-12 16:12:52,229 INFO flow.Flow (BaseFlow.java:logInfo(1378)) - [name] allocating threads: 1
2015-02-12 16:12:52,230 INFO flow.FlowStep (BaseFlowStep.java:logInfo(834)) - [name] starting step: (1/1) ...mblyhelpersplatform/multi
2015-02-12 16:12:52,230 INFO jvm.JvmMetrics (JvmMetrics.java:init(71)) - Cannot initialize JVM Metrics with processName=JobTracker, sessionId= - already initialized
2015-02-12 16:12:52,231 INFO jvm.JvmMetrics (JvmMetrics.java:init(71)) - Cannot initialize JVM Metrics with processName=JobTracker, sessionId= - already initialized
2015-02-12 16:12:52,248 INFO mapred.FileInputFormat (FileInputFormat.java:listStatus(247)) - Total input paths to process : 1
2015-02-12 16:12:52,265 INFO mapreduce.JobSubmitter (JobSubmitter.java:submitJobInternal(492)) - number of splits:1
2015-02-12 16:12:52,277 INFO mapreduce.JobSubmitter (JobSubmitter.java:printTokens(575)) - Submitting tokens for job: job_local1810252879_0026
2015-02-12 16:12:52,287 WARN conf.Configuration (Configuration.java:loadProperty(2510)) - file:/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/tmp/cascading/staging/soberoi1810252879/.staging/job_local1810252879_0026/job.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.retry.interval; Ignoring.
2015-02-12 16:12:52,287 WARN conf.Configuration (Configuration.java:loadProperty(2510)) - file:/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/tmp/cascading/staging/soberoi1810252879/.staging/job_local1810252879_0026/job.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.attempts; Ignoring.
2015-02-12 16:12:52,318 WARN conf.Configuration (Configuration.java:loadProperty(2510)) - file:/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/test/tmp/mapred/local/localRunner/soberoi/job_local1810252879_0026/job_local1810252879_0026.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.retry.interval; Ignoring.
2015-02-12 16:12:52,319 WARN conf.Configuration (Configuration.java:loadProperty(2510)) - file:/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/test/tmp/mapred/local/localRunner/soberoi/job_local1810252879_0026/job_local1810252879_0026.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.attempts; Ignoring.
2015-02-12 16:12:52,319 INFO mapreduce.Job (Job.java:submit(1299)) - The url to track the job: http://localhost:8080/
2015-02-12 16:12:52,320 INFO mapred.LocalJobRunner (LocalJobRunner.java:createOutputCommitter(471)) - OutputCommitter set in config null
2015-02-12 16:12:52,320 INFO flow.FlowStep (BaseFlowStep.java:logInfo(834)) - [name] submitted hadoop job: job_local1810252879_0026
2015-02-12 16:12:52,320 INFO mapred.LocalJobRunner (LocalJobRunner.java:createOutputCommitter(489)) - OutputCommitter is org.apache.hadoop.mapred.FileOutputCommitter
2015-02-12 16:12:52,320 INFO flow.FlowStep (BaseFlowStep.java:logInfo(834)) - [name] tracking url: http://localhost:8080/
2015-02-12 16:12:52,322 INFO mapred.LocalJobRunner (LocalJobRunner.java:runTasks(448)) - Waiting for map tasks
2015-02-12 16:12:52,322 INFO mapred.LocalJobRunner (LocalJobRunner.java:run(224)) - Starting task: attempt_local1810252879_0026_m_000000_0
2015-02-12 16:12:52,323 INFO util.ProcfsBasedProcessTree (ProcfsBasedProcessTree.java:isAvailable(181)) - ProcfsBasedProcessTree currently is supported only on Linux.
2015-02-12 16:12:52,323 INFO mapred.Task (Task.java:initialize(587)) - Using ResourceCalculatorProcessTree : null
2015-02-12 16:12:52,323 INFO io.MultiInputSplit (MultiInputSplit.java:readFields(161)) - current split input path: file:/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/classes/test/data/lhs.txt
2015-02-12 16:12:52,323 INFO mapred.MapTask (MapTask.java:updateJobWithSplit(482)) - Processing split: cascading.tap.hadoop.io.MultiInputSplit@66ce3c65
2015-02-12 16:12:52,324 INFO mapred.MapTask (MapTask.java:runOldMapper(436)) - numReduceTasks: 1
2015-02-12 16:12:52,339 INFO mapred.MapTask (MapTask.java:setEquator(1202)) - (EQUATOR) 0 kvi 26214396(104857584)
2015-02-12 16:12:52,340 INFO mapred.MapTask (MapTask.java:init(995)) - mapreduce.task.io.sort.mb: 100
2015-02-12 16:12:52,340 INFO mapred.MapTask (MapTask.java:init(996)) - soft limit at 83886080
2015-02-12 16:12:52,340 INFO mapred.MapTask (MapTask.java:init(997)) - bufstart = 0; bufvoid = 104857600
2015-02-12 16:12:52,340 INFO mapred.MapTask (MapTask.java:init(998)) - kvstart = 26214396; length = 6553600
2015-02-12 16:12:52,341 INFO mapred.MapTask (MapTask.java:createSortingCollector(402)) - Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer
2015-02-12 16:12:52,341 INFO hadoop.FlowMapper (FlowMapper.java:configure(71)) - cascading version: 2.6.3
2015-02-12 16:12:52,341 INFO hadoop.FlowMapper (FlowMapper.java:configure(72)) - child jvm opts: -Xmx200m
2015-02-12 16:12:52,347 INFO hadoop.FlowMapper (FlowMapper.java:configure(87)) - sourcing from: Hfs["TextDelimited[['num', 'char']]"]["/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/classes/test/data/lhs.txt"]
2015-02-12 16:12:52,348 INFO hadoop.FlowMapper (FlowMapper.java:configure(90)) - sinking to: GroupBy(name)[by:[{1}:'char']]
2015-02-12 16:12:52,348 INFO assembly.AggregateBy (AggregateBy.java:prepare(267)) - using threshold value: 2
2015-02-12 16:12:52,349 INFO assembly.AggregateBy (AggregateBy.java:removeEldestEntry(327)) - flushed keys num times: 1, with threshold: 2
2015-02-12 16:12:52,349 INFO assembly.AggregateBy (AggregateBy.java:removeEldestEntry(328)) - mem on flush (mb), free: 196, total: 512, max: 672
2015-02-12 16:12:52,350 INFO assembly.AggregateBy (AggregateBy.java:removeEldestEntry(333)) - total mem is 76% of max mem, to better utilize unused memory consider increasing current LRU threshold with system property "cascading.aggregateby.threshold"
2015-02-12 16:12:52,350 INFO assembly.AggregateBy (AggregateBy.java:removeEldestEntry(327)) - flushed keys num times: 3, with threshold: 2
2015-02-12 16:12:52,350 INFO assembly.AggregateBy (AggregateBy.java:removeEldestEntry(328)) - mem on flush (mb), free: 196, total: 512, max: 672
2015-02-12 16:12:52,351 INFO assembly.AggregateBy (AggregateBy.java:removeEldestEntry(333)) - total mem is 76% of max mem, to better utilize unused memory consider increasing current LRU threshold with system property "cascading.aggregateby.threshold"
2015-02-12 16:12:52,351 INFO assembly.AggregateBy (AggregateBy.java:removeEldestEntry(327)) - flushed keys num times: 5, with threshold: 2
2015-02-12 16:12:52,351 INFO assembly.AggregateBy (AggregateBy.java:removeEldestEntry(328)) - mem on flush (mb), free: 196, total: 512, max: 672
2015-02-12 16:12:52,351 INFO assembly.AggregateBy (AggregateBy.java:removeEldestEntry(333)) - total mem is 76% of max mem, to better utilize unused memory consider increasing current LRU threshold with system property "cascading.aggregateby.threshold"
2015-02-12 16:12:52,352 INFO assembly.AggregateBy (AggregateBy.java:removeEldestEntry(327)) - flushed keys num times: 7, with threshold: 2
2015-02-12 16:12:52,352 INFO assembly.AggregateBy (AggregateBy.java:removeEldestEntry(328)) - mem on flush (mb), free: 196, total: 512, max: 672
2015-02-12 16:12:52,352 INFO assembly.AggregateBy (AggregateBy.java:removeEldestEntry(333)) - total mem is 76% of max mem, to better utilize unused memory consider increasing current LRU threshold with system property "cascading.aggregateby.threshold"
2015-02-12 16:12:52,353 INFO mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) -
2015-02-12 16:12:52,353 INFO mapred.MapTask (MapTask.java:flush(1457)) - Starting flush of map output
2015-02-12 16:12:52,353 INFO mapred.MapTask (MapTask.java:flush(1475)) - Spilling map output
2015-02-12 16:12:52,353 INFO mapred.MapTask (MapTask.java:flush(1476)) - bufstart = 0; bufend = 432; bufvoid = 104857600
2015-02-12 16:12:52,354 INFO mapred.MapTask (MapTask.java:flush(1478)) - kvstart = 26214396(104857584); kvend = 26214364(104857456); length = 33/6553600
2015-02-12 16:12:52,356 INFO mapred.MapTask (MapTask.java:sortAndSpill(1660)) - Finished spill 0
2015-02-12 16:12:52,358 INFO mapred.Task (Task.java:done(1001)) - Task:attempt_local1810252879_0026_m_000000_0 is done. And is in the process of committing
2015-02-12 16:12:52,359 INFO mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) - file:/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/classes/test/data/lhs.txt:0+52
2015-02-12 16:12:52,359 INFO mapred.Task (Task.java:sendDone(1121)) - Task 'attempt_local1810252879_0026_m_000000_0' done.
2015-02-12 16:12:52,360 INFO mapred.LocalJobRunner (LocalJobRunner.java:run(249)) - Finishing task: attempt_local1810252879_0026_m_000000_0
2015-02-12 16:12:52,360 INFO mapred.LocalJobRunner (LocalJobRunner.java:runTasks(456)) - map task executor complete.
2015-02-12 16:12:52,360 INFO mapred.LocalJobRunner (LocalJobRunner.java:runTasks(448)) - Waiting for reduce tasks
2015-02-12 16:12:52,360 INFO mapred.LocalJobRunner (LocalJobRunner.java:run(302)) - Starting task: attempt_local1810252879_0026_r_000000_0
2015-02-12 16:12:52,362 INFO util.ProcfsBasedProcessTree (ProcfsBasedProcessTree.java:isAvailable(181)) - ProcfsBasedProcessTree currently is supported only on Linux.
2015-02-12 16:12:52,362 INFO mapred.Task (Task.java:initialize(587)) - Using ResourceCalculatorProcessTree : null
2015-02-12 16:12:52,362 INFO mapred.ReduceTask (ReduceTask.java:run(362)) - Using ShuffleConsumerPlugin: org.apache.hadoop.mapreduce.task.reduce.Shuffle@3ecf7dc2
2015-02-12 16:12:52,362 INFO reduce.MergeManagerImpl (MergeManagerImpl.java:<init>(195)) - MergerManager: memoryLimit=493250144, maxSingleShuffleLimit=123312536, mergeThreshold=325545120, ioSortFactor=10, memToMemMergeOutputsThreshold=10
2015-02-12 16:12:52,363 INFO reduce.EventFetcher (EventFetcher.java:run(61)) - attempt_local1810252879_0026_r_000000_0 Thread started: EventFetcher for fetching Map Completion Events
2015-02-12 16:12:52,364 INFO reduce.LocalFetcher (LocalFetcher.java:copyMapOutput(141)) - localfetcher#22 about to shuffle output of map attempt_local1810252879_0026_m_000000_0 decomp: 452 len: 456 to MEMORY
2015-02-12 16:12:52,364 INFO reduce.InMemoryMapOutput (InMemoryMapOutput.java:shuffle(100)) - Read 452 bytes from map-output for attempt_local1810252879_0026_m_000000_0
2015-02-12 16:12:52,364 INFO reduce.MergeManagerImpl (MergeManagerImpl.java:closeInMemoryFile(313)) - closeInMemoryFile -> map-output of size: 452, inMemoryMapOutputs.size() -> 1, commitMemory -> 0, usedMemory ->452
2015-02-12 16:12:52,365 INFO reduce.EventFetcher (EventFetcher.java:run(76)) - EventFetcher is interrupted.. Returning
2015-02-12 16:12:52,365 INFO mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) - 1 / 1 copied.
2015-02-12 16:12:52,366 INFO reduce.MergeManagerImpl (MergeManagerImpl.java:finalMerge(673)) - finalMerge called with 1 in-memory map-outputs and 0 on-disk map-outputs
2015-02-12 16:12:52,367 INFO mapred.Merger (Merger.java:merge(606)) - Merging 1 sorted segments
2015-02-12 16:12:52,368 INFO mapred.Merger (Merger.java:merge(705)) - Down to the last merge-pass, with 1 segments left of total size: 436 bytes
2015-02-12 16:12:52,369 INFO reduce.MergeManagerImpl (MergeManagerImpl.java:finalMerge(750)) - Merged 1 segments, 452 bytes to disk to satisfy reduce memory limit
2015-02-12 16:12:52,369 INFO reduce.MergeManagerImpl (MergeManagerImpl.java:finalMerge(780)) - Merging 1 files, 456 bytes from disk
2015-02-12 16:12:52,369 INFO reduce.MergeManagerImpl (MergeManagerImpl.java:finalMerge(795)) - Merging 0 segments, 0 bytes from memory into reduce
2015-02-12 16:12:52,369 INFO mapred.Merger (Merger.java:merge(606)) - Merging 1 sorted segments
2015-02-12 16:12:52,370 INFO mapred.Merger (Merger.java:merge(705)) - Down to the last merge-pass, with 1 segments left of total size: 436 bytes
2015-02-12 16:12:52,370 INFO mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) - 1 / 1 copied.
2015-02-12 16:12:52,370 INFO hadoop.FlowReducer (FlowReducer.java:configure(78)) - cascading version: 2.6.3
2015-02-12 16:12:52,370 INFO hadoop.FlowReducer (FlowReducer.java:configure(79)) - child jvm opts: -Xmx200m
2015-02-12 16:12:52,377 INFO hadoop.FlowReducer (FlowReducer.java:configure(97)) - sourcing from: GroupBy(name)[by:[{1}:'char']]
2015-02-12 16:12:52,377 INFO hadoop.FlowReducer (FlowReducer.java:configure(100)) - sinking to: Hfs["TextDelimited[['char', 'sum', 'count', 'average', 'average2', 'first']]"]["/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/test/output/hadoop2-mr1/assemblyhelpersplatform/multi"]
2015-02-12 16:12:52,385 INFO mapred.Task (Task.java:done(1001)) - Task:attempt_local1810252879_0026_r_000000_0 is done. And is in the process of committing
2015-02-12 16:12:52,386 INFO mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) - 1 / 1 copied.
2015-02-12 16:12:52,386 INFO mapred.Task (Task.java:commit(1162)) - Task attempt_local1810252879_0026_r_000000_0 is allowed to commit now
2015-02-12 16:12:52,386 INFO output.FileOutputCommitter (FileOutputCommitter.java:commitTask(439)) - Saved output of task 'attempt_local1810252879_0026_r_000000_0' to file:/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/test/output/hadoop2-mr1/assemblyhelpersplatform/multi/_temporary/0/task_local1810252879_0026_r_000000
2015-02-12 16:12:52,387 INFO mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) - reduce > reduce
2015-02-12 16:12:52,387 INFO mapred.Task (Task.java:sendDone(1121)) - Task 'attempt_local1810252879_0026_r_000000_0' done.
2015-02-12 16:12:52,387 INFO mapred.LocalJobRunner (LocalJobRunner.java:run(325)) - Finishing task: attempt_local1810252879_0026_r_000000_0
2015-02-12 16:12:52,387 INFO mapred.LocalJobRunner (LocalJobRunner.java:runTasks(456)) - reduce task executor complete.
2015-02-12 16:12:52,404 INFO util.Hadoop18TapUtil (Hadoop18TapUtil.java:cleanTempPath(219)) - deleting temp path /Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/test/output/hadoop2-mr1/assemblyhelpersplatform/multi/_temporary
2015-02-12 16:12:52,423 INFO mapred.FileInputFormat (FileInputFormat.java:listStatus(247)) - Total input paths to process : 1
2015-02-12 16:12:52,445 INFO mapred.FileInputFormat (FileInputFormat.java:listStatus(247)) - Total input paths to process : 1
2015-02-12 16:12:52,446 INFO cascading.PlatformTestCase (PlatformTestCase.java:tearDown(209)) - copying to local /Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/test/output//hadoop2-mr1/assemblyhelpersplatform/multi
2015-02-12 16:12:52,467 INFO util.HadoopUtil (HadoopUtil.java:findMainClass(336)) - using default application jar, may cause class not found exceptions on the cluster
2015-02-12 16:12:52,467 INFO planner.HadoopPlanner (HadoopPlanner.java:initialize(225)) - using application jar: /Users/soberoi/.gradle/caches/modules-2/files-2.1/cascading/cascading-hadoop2-mr1/2.6.3/9ae016e2b8d246b9c1de9168e715d9f1b971aa4a/cascading-hadoop2-mr1-2.6.3.jar
2015-02-12 16:12:52,470 INFO Configuration.deprecation (Configuration.java:warnOnceIfDeprecated(1022)) - mapred.used.genericoptionsparser is deprecated. Instead, use mapreduce.client.genericoptionsparser.used
2015-02-12 16:12:52,471 INFO Configuration.deprecation (Configuration.java:warnOnceIfDeprecated(1022)) - mapred.job.tracker is deprecated. Instead, use mapreduce.jobtracker.address
2015-02-12 16:12:52,472 INFO Configuration.deprecation (Configuration.java:warnOnceIfDeprecated(1022)) - mapred.output.compress is deprecated. Instead, use mapreduce.output.fileoutputformat.compress
2015-02-12 16:12:52,475 INFO flow.Flow (BaseFlow.java:logInfo(1378)) - [shape] starting
2015-02-12 16:12:52,475 INFO flow.Flow (BaseFlow.java:logInfo(1378)) - [shape] source: Hfs["TextLine[['offset', 'line']->[ALL]]"]["/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/classes/test/data/lower.txt"]
2015-02-12 16:12:52,475 INFO flow.Flow (BaseFlow.java:logInfo(1378)) - [shape] sink: Hfs["TextLine[['num']]"]["/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/test/output/hadoop2-mr1/assemblyhelpersplatform/retainnarrow"]
2015-02-12 16:12:52,475 INFO flow.Flow (BaseFlow.java:logInfo(1378)) - [shape] parallel execution is enabled: false
2015-02-12 16:12:52,475 INFO flow.Flow (BaseFlow.java:logInfo(1378)) - [shape] starting jobs: 1
2015-02-12 16:12:52,475 INFO flow.Flow (BaseFlow.java:logInfo(1378)) - [shape] allocating threads: 1
2015-02-12 16:12:52,476 INFO flow.FlowStep (BaseFlowStep.java:logInfo(834)) - [shape] starting step: (1/1) ...persplatform/retainnarrow
2015-02-12 16:12:52,476 INFO jvm.JvmMetrics (JvmMetrics.java:init(71)) - Cannot initialize JVM Metrics with processName=JobTracker, sessionId= - already initialized
2015-02-12 16:12:52,477 INFO jvm.JvmMetrics (JvmMetrics.java:init(71)) - Cannot initialize JVM Metrics with processName=JobTracker, sessionId= - already initialized
2015-02-12 16:12:52,494 INFO mapred.FileInputFormat (FileInputFormat.java:listStatus(247)) - Total input paths to process : 1
2015-02-12 16:12:52,510 INFO mapreduce.JobSubmitter (JobSubmitter.java:submitJobInternal(492)) - number of splits:1
2015-02-12 16:12:52,521 INFO mapreduce.JobSubmitter (JobSubmitter.java:printTokens(575)) - Submitting tokens for job: job_local783720211_0027
2015-02-12 16:12:52,532 WARN conf.Configuration (Configuration.java:loadProperty(2510)) - file:/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/tmp/cascading/staging/soberoi783720211/.staging/job_local783720211_0027/job.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.retry.interval; Ignoring.
2015-02-12 16:12:52,533 WARN conf.Configuration (Configuration.java:loadProperty(2510)) - file:/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/tmp/cascading/staging/soberoi783720211/.staging/job_local783720211_0027/job.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.attempts; Ignoring.
2015-02-12 16:12:52,560 WARN conf.Configuration (Configuration.java:loadProperty(2510)) - file:/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/test/tmp/mapred/local/localRunner/soberoi/job_local783720211_0027/job_local783720211_0027.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.retry.interval; Ignoring.
2015-02-12 16:12:52,560 WARN conf.Configuration (Configuration.java:loadProperty(2510)) - file:/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/test/tmp/mapred/local/localRunner/soberoi/job_local783720211_0027/job_local783720211_0027.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.attempts; Ignoring.
2015-02-12 16:12:52,561 INFO mapreduce.Job (Job.java:submit(1299)) - The url to track the job: http://localhost:8080/
2015-02-12 16:12:52,561 INFO mapred.LocalJobRunner (LocalJobRunner.java:createOutputCommitter(471)) - OutputCommitter set in config null
2015-02-12 16:12:52,561 INFO flow.FlowStep (BaseFlowStep.java:logInfo(834)) - [shape] submitted hadoop job: job_local783720211_0027
2015-02-12 16:12:52,561 INFO mapred.LocalJobRunner (LocalJobRunner.java:createOutputCommitter(489)) - OutputCommitter is org.apache.hadoop.mapred.FileOutputCommitter
2015-02-12 16:12:52,562 INFO flow.FlowStep (BaseFlowStep.java:logInfo(834)) - [shape] tracking url: http://localhost:8080/
2015-02-12 16:12:52,563 INFO mapred.LocalJobRunner (LocalJobRunner.java:runTasks(448)) - Waiting for map tasks
2015-02-12 16:12:52,563 INFO mapred.LocalJobRunner (LocalJobRunner.java:run(224)) - Starting task: attempt_local783720211_0027_m_000000_0
2015-02-12 16:12:52,565 INFO util.ProcfsBasedProcessTree (ProcfsBasedProcessTree.java:isAvailable(181)) - ProcfsBasedProcessTree currently is supported only on Linux.
2015-02-12 16:12:52,565 INFO mapred.Task (Task.java:initialize(587)) - Using ResourceCalculatorProcessTree : null
2015-02-12 16:12:52,566 INFO io.MultiInputSplit (MultiInputSplit.java:readFields(161)) - current split input path: file:/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/classes/test/data/lower.txt
2015-02-12 16:12:52,566 INFO mapred.MapTask (MapTask.java:updateJobWithSplit(482)) - Processing split: cascading.tap.hadoop.io.MultiInputSplit@2fe6f732
2015-02-12 16:12:52,566 INFO mapred.MapTask (MapTask.java:runOldMapper(436)) - numReduceTasks: 0
2015-02-12 16:12:52,572 INFO hadoop.FlowMapper (FlowMapper.java:configure(71)) - cascading version: 2.6.3
2015-02-12 16:12:52,572 INFO hadoop.FlowMapper (FlowMapper.java:configure(72)) - child jvm opts: -Xmx200m
2015-02-12 16:12:52,574 INFO hadoop.FlowMapper (FlowMapper.java:configure(87)) - sourcing from: Hfs["TextLine[['offset', 'line']->[ALL]]"]["/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/classes/test/data/lower.txt"]
2015-02-12 16:12:52,575 INFO hadoop.FlowMapper (FlowMapper.java:configure(90)) - sinking to: Hfs["TextLine[['num']]"]["/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/test/output/hadoop2-mr1/assemblyhelpersplatform/retainnarrow"]
2015-02-12 16:12:52,576 INFO mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) -
2015-02-12 16:12:52,577 INFO mapred.Task (Task.java:done(1001)) - Task:attempt_local783720211_0027_m_000000_0 is done. And is in the process of committing
2015-02-12 16:12:52,578 INFO mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) -
2015-02-12 16:12:52,578 INFO mapred.Task (Task.java:commit(1162)) - Task attempt_local783720211_0027_m_000000_0 is allowed to commit now
2015-02-12 16:12:52,578 INFO output.FileOutputCommitter (FileOutputCommitter.java:commitTask(439)) - Saved output of task 'attempt_local783720211_0027_m_000000_0' to file:/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/test/output/hadoop2-mr1/assemblyhelpersplatform/retainnarrow/_temporary/0/task_local783720211_0027_m_000000
2015-02-12 16:12:52,579 INFO mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) - file:/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/classes/test/data/lower.txt:0+19
2015-02-12 16:12:52,579 INFO mapred.Task (Task.java:sendDone(1121)) - Task 'attempt_local783720211_0027_m_000000_0' done.
2015-02-12 16:12:52,579 INFO mapred.LocalJobRunner (LocalJobRunner.java:run(249)) - Finishing task: attempt_local783720211_0027_m_000000_0
2015-02-12 16:12:52,579 INFO mapred.LocalJobRunner (LocalJobRunner.java:runTasks(456)) - map task executor complete.
2015-02-12 16:12:52,603 INFO util.Hadoop18TapUtil (Hadoop18TapUtil.java:cleanTempPath(219)) - deleting temp path /Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/test/output/hadoop2-mr1/assemblyhelpersplatform/retainnarrow/_temporary
2015-02-12 16:12:52,624 INFO mapred.FileInputFormat (FileInputFormat.java:listStatus(247)) - Total input paths to process : 1
2015-02-12 16:12:52,625 INFO cascading.PlatformTestCase (PlatformTestCase.java:tearDown(209)) - copying to local /Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/test/output//hadoop2-mr1/assemblyhelpersplatform/retainnarrow
2015-02-12 16:12:52,636 INFO util.HadoopUtil (HadoopUtil.java:findMainClass(336)) - using default application jar, may cause class not found exceptions on the cluster
2015-02-12 16:12:52,636 INFO planner.HadoopPlanner (HadoopPlanner.java:initialize(225)) - using application jar: /Users/soberoi/.gradle/caches/modules-2/files-2.1/cascading/cascading-hadoop2-mr1/2.6.3/9ae016e2b8d246b9c1de9168e715d9f1b971aa4a/cascading-hadoop2-mr1-2.6.3.jar
2015-02-12 16:12:52,639 INFO Configuration.deprecation (Configuration.java:warnOnceIfDeprecated(1022)) - mapred.used.genericoptionsparser is deprecated. Instead, use mapreduce.client.genericoptionsparser.used
2015-02-12 16:12:52,640 INFO Configuration.deprecation (Configuration.java:warnOnceIfDeprecated(1022)) - mapred.job.tracker is deprecated. Instead, use mapreduce.jobtracker.address
2015-02-12 16:12:52,640 INFO Configuration.deprecation (Configuration.java:warnOnceIfDeprecated(1022)) - mapred.output.compress is deprecated. Instead, use mapreduce.output.fileoutputformat.compress
2015-02-12 16:12:52,643 INFO flow.Flow (BaseFlow.java:logInfo(1378)) - [shape] starting
2015-02-12 16:12:52,643 INFO flow.Flow (BaseFlow.java:logInfo(1378)) - [shape] source: Hfs["TextLine[['offset', 'line']->[ALL]]"]["/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/classes/test/data/lower.txt"]
2015-02-12 16:12:52,643 INFO flow.Flow (BaseFlow.java:logInfo(1378)) - [shape] sink: Hfs["TextLine[['item']->['char', 'item']]"]["/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/test/output/hadoop2-mr1/assemblyhelpersplatform/renamenarrow"]
2015-02-12 16:12:52,643 INFO flow.Flow (BaseFlow.java:logInfo(1378)) - [shape] parallel execution is enabled: false
2015-02-12 16:12:52,644 INFO flow.Flow (BaseFlow.java:logInfo(1378)) - [shape] starting jobs: 1
2015-02-12 16:12:52,644 INFO flow.Flow (BaseFlow.java:logInfo(1378)) - [shape] allocating threads: 1
2015-02-12 16:12:52,644 INFO flow.FlowStep (BaseFlowStep.java:logInfo(834)) - [shape] starting step: (1/1) ...persplatform/renamenarrow
2015-02-12 16:12:52,645 INFO jvm.JvmMetrics (JvmMetrics.java:init(71)) - Cannot initialize JVM Metrics with processName=JobTracker, sessionId= - already initialized
2015-02-12 16:12:52,645 INFO jvm.JvmMetrics (JvmMetrics.java:init(71)) - Cannot initialize JVM Metrics with processName=JobTracker, sessionId= - already initialized
2015-02-12 16:12:52,663 INFO mapred.FileInputFormat (FileInputFormat.java:listStatus(247)) - Total input paths to process : 1
2015-02-12 16:12:52,680 INFO mapreduce.JobSubmitter (JobSubmitter.java:submitJobInternal(492)) - number of splits:1
2015-02-12 16:12:52,691 INFO mapreduce.JobSubmitter (JobSubmitter.java:printTokens(575)) - Submitting tokens for job: job_local1949352354_0028
2015-02-12 16:12:52,702 WARN conf.Configuration (Configuration.java:loadProperty(2510)) - file:/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/tmp/cascading/staging/soberoi1949352354/.staging/job_local1949352354_0028/job.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.retry.interval; Ignoring.
2015-02-12 16:12:52,703 WARN conf.Configuration (Configuration.java:loadProperty(2510)) - file:/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/tmp/cascading/staging/soberoi1949352354/.staging/job_local1949352354_0028/job.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.attempts; Ignoring.
2015-02-12 16:12:52,730 WARN conf.Configuration (Configuration.java:loadProperty(2510)) - file:/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/test/tmp/mapred/local/localRunner/soberoi/job_local1949352354_0028/job_local1949352354_0028.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.retry.interval; Ignoring.
2015-02-12 16:12:52,731 WARN conf.Configuration (Configuration.java:loadProperty(2510)) - file:/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/test/tmp/mapred/local/localRunner/soberoi/job_local1949352354_0028/job_local1949352354_0028.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.attempts; Ignoring.
2015-02-12 16:12:52,732 INFO mapreduce.Job (Job.java:submit(1299)) - The url to track the job: http://localhost:8080/
2015-02-12 16:12:52,732 INFO mapred.LocalJobRunner (LocalJobRunner.java:createOutputCommitter(471)) - OutputCommitter set in config null
2015-02-12 16:12:52,732 INFO flow.FlowStep (BaseFlowStep.java:logInfo(834)) - [shape] submitted hadoop job: job_local1949352354_0028
2015-02-12 16:12:52,732 INFO mapred.LocalJobRunner (LocalJobRunner.java:createOutputCommitter(489)) - OutputCommitter is org.apache.hadoop.mapred.FileOutputCommitter
2015-02-12 16:12:52,732 INFO flow.FlowStep (BaseFlowStep.java:logInfo(834)) - [shape] tracking url: http://localhost:8080/
2015-02-12 16:12:52,734 INFO mapred.LocalJobRunner (LocalJobRunner.java:runTasks(448)) - Waiting for map tasks
2015-02-12 16:12:52,734 INFO mapred.LocalJobRunner (LocalJobRunner.java:run(224)) - Starting task: attempt_local1949352354_0028_m_000000_0
2015-02-12 16:12:52,735 INFO util.ProcfsBasedProcessTree (ProcfsBasedProcessTree.java:isAvailable(181)) - ProcfsBasedProcessTree currently is supported only on Linux.
2015-02-12 16:12:52,735 INFO mapred.Task (Task.java:initialize(587)) - Using ResourceCalculatorProcessTree : null
2015-02-12 16:12:52,736 INFO io.MultiInputSplit (MultiInputSplit.java:readFields(161)) - current split input path: file:/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/classes/test/data/lower.txt
2015-02-12 16:12:52,736 INFO mapred.MapTask (MapTask.java:updateJobWithSplit(482)) - Processing split: cascading.tap.hadoop.io.MultiInputSplit@2d3c2ba1
2015-02-12 16:12:52,736 INFO mapred.MapTask (MapTask.java:runOldMapper(436)) - numReduceTasks: 0
2015-02-12 16:12:52,741 INFO hadoop.FlowMapper (FlowMapper.java:configure(71)) - cascading version: 2.6.3
2015-02-12 16:12:52,742 INFO hadoop.FlowMapper (FlowMapper.java:configure(72)) - child jvm opts: -Xmx200m
2015-02-12 16:12:52,744 INFO hadoop.FlowMapper (FlowMapper.java:configure(87)) - sourcing from: Hfs["TextLine[['offset', 'line']->[ALL]]"]["/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/classes/test/data/lower.txt"]
2015-02-12 16:12:52,744 INFO hadoop.FlowMapper (FlowMapper.java:configure(90)) - sinking to: Hfs["TextLine[['item']->['char', 'item']]"]["/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/test/output/hadoop2-mr1/assemblyhelpersplatform/renamenarrow"]
2015-02-12 16:12:52,746 INFO mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) -
2015-02-12 16:12:52,746 INFO mapred.Task (Task.java:done(1001)) - Task:attempt_local1949352354_0028_m_000000_0 is done. And is in the process of committing
2015-02-12 16:12:52,747 INFO mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) -
2015-02-12 16:12:52,748 INFO mapred.Task (Task.java:commit(1162)) - Task attempt_local1949352354_0028_m_000000_0 is allowed to commit now
2015-02-12 16:12:52,748 INFO output.FileOutputCommitter (FileOutputCommitter.java:commitTask(439)) - Saved output of task 'attempt_local1949352354_0028_m_000000_0' to file:/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/test/output/hadoop2-mr1/assemblyhelpersplatform/renamenarrow/_temporary/0/task_local1949352354_0028_m_000000
2015-02-12 16:12:52,749 INFO mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) - file:/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/classes/test/data/lower.txt:0+19
2015-02-12 16:12:52,749 INFO mapred.Task (Task.java:sendDone(1121)) - Task 'attempt_local1949352354_0028_m_000000_0' done.
2015-02-12 16:12:52,749 INFO mapred.LocalJobRunner (LocalJobRunner.java:run(249)) - Finishing task: attempt_local1949352354_0028_m_000000_0
2015-02-12 16:12:52,749 INFO mapred.LocalJobRunner (LocalJobRunner.java:runTasks(456)) - map task executor complete.
2015-02-12 16:12:52,758 INFO util.Hadoop18TapUtil (Hadoop18TapUtil.java:cleanTempPath(219)) - deleting temp path /Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/test/output/hadoop2-mr1/assemblyhelpersplatform/renamenarrow/_temporary
2015-02-12 16:12:52,779 INFO mapred.FileInputFormat (FileInputFormat.java:listStatus(247)) - Total input paths to process : 1
2015-02-12 16:12:52,780 INFO cascading.PlatformTestCase (PlatformTestCase.java:tearDown(209)) - copying to local /Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/test/output//hadoop2-mr1/assemblyhelpersplatform/renamenarrow
2015-02-12 16:12:52,792 INFO util.HadoopUtil (HadoopUtil.java:findMainClass(336)) - using default application jar, may cause class not found exceptions on the cluster
2015-02-12 16:12:52,792 INFO planner.HadoopPlanner (HadoopPlanner.java:initialize(225)) - using application jar: /Users/soberoi/.gradle/caches/modules-2/files-2.1/cascading/cascading-hadoop2-mr1/2.6.3/9ae016e2b8d246b9c1de9168e715d9f1b971aa4a/cascading-hadoop2-mr1-2.6.3.jar
2015-02-12 16:12:52,795 INFO Configuration.deprecation (Configuration.java:warnOnceIfDeprecated(1022)) - mapred.used.genericoptionsparser is deprecated. Instead, use mapreduce.client.genericoptionsparser.used
2015-02-12 16:12:52,796 INFO Configuration.deprecation (Configuration.java:warnOnceIfDeprecated(1022)) - mapred.job.tracker is deprecated. Instead, use mapreduce.jobtracker.address
2015-02-12 16:12:52,797 INFO Configuration.deprecation (Configuration.java:warnOnceIfDeprecated(1022)) - mapred.output.compress is deprecated. Instead, use mapreduce.output.fileoutputformat.compress
2015-02-12 16:12:52,800 INFO flow.Flow (BaseFlow.java:logInfo(1378)) - [average] starting
2015-02-12 16:12:52,800 INFO flow.Flow (BaseFlow.java:logInfo(1378)) - [average] source: Hfs["TextDelimited[['num', 'char']]"]["/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/classes/test/data/lhs.txt"]
2015-02-12 16:12:52,800 INFO flow.Flow (BaseFlow.java:logInfo(1378)) - [average] sink: Hfs["TextDelimited[['char', 'average']]"]["/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/test/output/hadoop2-mr1/assemblyhelpersplatform/averagenull"]
2015-02-12 16:12:52,800 INFO flow.Flow (BaseFlow.java:logInfo(1378)) - [average] parallel execution is enabled: false
2015-02-12 16:12:52,801 INFO flow.Flow (BaseFlow.java:logInfo(1378)) - [average] starting jobs: 1
2015-02-12 16:12:52,801 INFO flow.Flow (BaseFlow.java:logInfo(1378)) - [average] allocating threads: 1
2015-02-12 16:12:52,801 INFO flow.FlowStep (BaseFlowStep.java:logInfo(834)) - [average] starting step: (1/1) ...lpersplatform/averagenull
2015-02-12 16:12:52,801 INFO jvm.JvmMetrics (JvmMetrics.java:init(71)) - Cannot initialize JVM Metrics with processName=JobTracker, sessionId= - already initialized
2015-02-12 16:12:52,802 INFO jvm.JvmMetrics (JvmMetrics.java:init(71)) - Cannot initialize JVM Metrics with processName=JobTracker, sessionId= - already initialized
2015-02-12 16:12:52,820 INFO mapred.FileInputFormat (FileInputFormat.java:listStatus(247)) - Total input paths to process : 1
2015-02-12 16:12:52,836 INFO mapreduce.JobSubmitter (JobSubmitter.java:submitJobInternal(492)) - number of splits:1
2015-02-12 16:12:52,852 INFO mapreduce.JobSubmitter (JobSubmitter.java:printTokens(575)) - Submitting tokens for job: job_local1666493537_0029
2015-02-12 16:12:52,863 WARN conf.Configuration (Configuration.java:loadProperty(2510)) - file:/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/tmp/cascading/staging/soberoi1666493537/.staging/job_local1666493537_0029/job.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.retry.interval; Ignoring.
2015-02-12 16:12:52,864 WARN conf.Configuration (Configuration.java:loadProperty(2510)) - file:/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/tmp/cascading/staging/soberoi1666493537/.staging/job_local1666493537_0029/job.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.attempts; Ignoring.
2015-02-12 16:12:52,892 WARN conf.Configuration (Configuration.java:loadProperty(2510)) - file:/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/test/tmp/mapred/local/localRunner/soberoi/job_local1666493537_0029/job_local1666493537_0029.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.retry.interval; Ignoring.
2015-02-12 16:12:52,893 WARN conf.Configuration (Configuration.java:loadProperty(2510)) - file:/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/test/tmp/mapred/local/localRunner/soberoi/job_local1666493537_0029/job_local1666493537_0029.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.attempts; Ignoring.
2015-02-12 16:12:52,894 INFO mapreduce.Job (Job.java:submit(1299)) - The url to track the job: http://localhost:8080/
2015-02-12 16:12:52,894 INFO mapred.LocalJobRunner (LocalJobRunner.java:createOutputCommitter(471)) - OutputCommitter set in config null
2015-02-12 16:12:52,894 INFO flow.FlowStep (BaseFlowStep.java:logInfo(834)) - [average] submitted hadoop job: job_local1666493537_0029
2015-02-12 16:12:52,894 INFO mapred.LocalJobRunner (LocalJobRunner.java:createOutputCommitter(489)) - OutputCommitter is org.apache.hadoop.mapred.FileOutputCommitter
2015-02-12 16:12:52,894 INFO flow.FlowStep (BaseFlowStep.java:logInfo(834)) - [average] tracking url: http://localhost:8080/
2015-02-12 16:12:52,896 INFO mapred.LocalJobRunner (LocalJobRunner.java:runTasks(448)) - Waiting for map tasks
2015-02-12 16:12:52,896 INFO mapred.LocalJobRunner (LocalJobRunner.java:run(224)) - Starting task: attempt_local1666493537_0029_m_000000_0
2015-02-12 16:12:52,897 INFO util.ProcfsBasedProcessTree (ProcfsBasedProcessTree.java:isAvailable(181)) - ProcfsBasedProcessTree currently is supported only on Linux.
2015-02-12 16:12:52,897 INFO mapred.Task (Task.java:initialize(587)) - Using ResourceCalculatorProcessTree : null
2015-02-12 16:12:52,898 INFO io.MultiInputSplit (MultiInputSplit.java:readFields(161)) - current split input path: file:/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/classes/test/data/lhs.txt
2015-02-12 16:12:52,898 INFO mapred.MapTask (MapTask.java:updateJobWithSplit(482)) - Processing split: cascading.tap.hadoop.io.MultiInputSplit@1d4a0efb
2015-02-12 16:12:52,898 INFO mapred.MapTask (MapTask.java:runOldMapper(436)) - numReduceTasks: 1
2015-02-12 16:12:52,909 INFO mapred.MapTask (MapTask.java:setEquator(1202)) - (EQUATOR) 0 kvi 26214396(104857584)
2015-02-12 16:12:52,910 INFO mapred.MapTask (MapTask.java:init(995)) - mapreduce.task.io.sort.mb: 100
2015-02-12 16:12:52,910 INFO mapred.MapTask (MapTask.java:init(996)) - soft limit at 83886080
2015-02-12 16:12:52,910 INFO mapred.MapTask (MapTask.java:init(997)) - bufstart = 0; bufvoid = 104857600
2015-02-12 16:12:52,910 INFO mapred.MapTask (MapTask.java:init(998)) - kvstart = 26214396; length = 6553600
2015-02-12 16:12:52,911 INFO mapred.MapTask (MapTask.java:createSortingCollector(402)) - Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer
2015-02-12 16:12:52,911 INFO hadoop.FlowMapper (FlowMapper.java:configure(71)) - cascading version: 2.6.3
2015-02-12 16:12:52,911 INFO hadoop.FlowMapper (FlowMapper.java:configure(72)) - child jvm opts: -Xmx200m
2015-02-12 16:12:52,914 INFO hadoop.FlowMapper (FlowMapper.java:configure(87)) - sourcing from: Hfs["TextDelimited[['num', 'char']]"]["/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/classes/test/data/lhs.txt"]
2015-02-12 16:12:52,914 INFO hadoop.FlowMapper (FlowMapper.java:configure(90)) - sinking to: GroupBy(average)[by:[{1}:'char']]
2015-02-12 16:12:52,914 INFO assembly.AggregateBy (AggregateBy.java:prepare(267)) - using threshold value: 2
2015-02-12 16:12:52,917 INFO assembly.AggregateBy (AggregateBy.java:removeEldestEntry(327)) - flushed keys num times: 1, with threshold: 2
2015-02-12 16:12:52,917 INFO assembly.AggregateBy (AggregateBy.java:removeEldestEntry(328)) - mem on flush (mb), free: 172, total: 513, max: 672
2015-02-12 16:12:52,917 INFO assembly.AggregateBy (AggregateBy.java:removeEldestEntry(333)) - total mem is 76% of max mem, to better utilize unused memory consider increasing current LRU threshold with system property "cascading.aggregateby.threshold"
2015-02-12 16:12:52,918 INFO assembly.AggregateBy (AggregateBy.java:removeEldestEntry(327)) - flushed keys num times: 3, with threshold: 2
2015-02-12 16:12:52,918 INFO assembly.AggregateBy (AggregateBy.java:removeEldestEntry(328)) - mem on flush (mb), free: 172, total: 513, max: 672
2015-02-12 16:12:52,918 INFO assembly.AggregateBy (AggregateBy.java:removeEldestEntry(333)) - total mem is 76% of max mem, to better utilize unused memory consider increasing current LRU threshold with system property "cascading.aggregateby.threshold"
2015-02-12 16:12:52,918 INFO assembly.AggregateBy (AggregateBy.java:removeEldestEntry(327)) - flushed keys num times: 5, with threshold: 2
2015-02-12 16:12:52,918 INFO assembly.AggregateBy (AggregateBy.java:removeEldestEntry(328)) - mem on flush (mb), free: 172, total: 513, max: 672
2015-02-12 16:12:52,918 INFO assembly.AggregateBy (AggregateBy.java:removeEldestEntry(333)) - total mem is 76% of max mem, to better utilize unused memory consider increasing current LRU threshold with system property "cascading.aggregateby.threshold"
2015-02-12 16:12:52,919 INFO assembly.AggregateBy (AggregateBy.java:removeEldestEntry(327)) - flushed keys num times: 7, with threshold: 2
2015-02-12 16:12:52,919 INFO assembly.AggregateBy (AggregateBy.java:removeEldestEntry(328)) - mem on flush (mb), free: 172, total: 513, max: 672
2015-02-12 16:12:52,919 INFO assembly.AggregateBy (AggregateBy.java:removeEldestEntry(333)) - total mem is 76% of max mem, to better utilize unused memory consider increasing current LRU threshold with system property "cascading.aggregateby.threshold"
2015-02-12 16:12:52,920 INFO mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) -
2015-02-12 16:12:52,920 INFO mapred.MapTask (MapTask.java:flush(1457)) - Starting flush of map output
2015-02-12 16:12:52,920 INFO mapred.MapTask (MapTask.java:flush(1475)) - Spilling map output
2015-02-12 16:12:52,920 INFO mapred.MapTask (MapTask.java:flush(1476)) - bufstart = 0; bufend = 180; bufvoid = 104857600
2015-02-12 16:12:52,920 INFO mapred.MapTask (MapTask.java:flush(1478)) - kvstart = 26214396(104857584); kvend = 26214364(104857456); length = 33/6553600
2015-02-12 16:12:52,922 INFO mapred.MapTask (MapTask.java:sortAndSpill(1660)) - Finished spill 0
2015-02-12 16:12:52,923 INFO mapred.Task (Task.java:done(1001)) - Task:attempt_local1666493537_0029_m_000000_0 is done. And is in the process of committing
2015-02-12 16:12:52,924 INFO mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) - file:/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/classes/test/data/lhs.txt:0+52
2015-02-12 16:12:52,924 INFO mapred.Task (Task.java:sendDone(1121)) - Task 'attempt_local1666493537_0029_m_000000_0' done.
2015-02-12 16:12:52,924 INFO mapred.LocalJobRunner (LocalJobRunner.java:run(249)) - Finishing task: attempt_local1666493537_0029_m_000000_0
2015-02-12 16:12:52,924 INFO mapred.LocalJobRunner (LocalJobRunner.java:runTasks(456)) - map task executor complete.
2015-02-12 16:12:52,925 INFO mapred.LocalJobRunner (LocalJobRunner.java:runTasks(448)) - Waiting for reduce tasks
2015-02-12 16:12:52,925 INFO mapred.LocalJobRunner (LocalJobRunner.java:run(302)) - Starting task: attempt_local1666493537_0029_r_000000_0
2015-02-12 16:12:52,926 INFO util.ProcfsBasedProcessTree (ProcfsBasedProcessTree.java:isAvailable(181)) - ProcfsBasedProcessTree currently is supported only on Linux.
2015-02-12 16:12:52,926 INFO mapred.Task (Task.java:initialize(587)) - Using ResourceCalculatorProcessTree : null
2015-02-12 16:12:52,926 INFO mapred.ReduceTask (ReduceTask.java:run(362)) - Using ShuffleConsumerPlugin: org.apache.hadoop.mapreduce.task.reduce.Shuffle@40542fa7
2015-02-12 16:12:52,926 INFO reduce.MergeManagerImpl (MergeManagerImpl.java:<init>(195)) - MergerManager: memoryLimit=493250144, maxSingleShuffleLimit=123312536, mergeThreshold=325545120, ioSortFactor=10, memToMemMergeOutputsThreshold=10
2015-02-12 16:12:52,927 INFO reduce.EventFetcher (EventFetcher.java:run(61)) - attempt_local1666493537_0029_r_000000_0 Thread started: EventFetcher for fetching Map Completion Events
2015-02-12 16:12:52,928 INFO reduce.LocalFetcher (LocalFetcher.java:copyMapOutput(141)) - localfetcher#23 about to shuffle output of map attempt_local1666493537_0029_m_000000_0 decomp: 200 len: 204 to MEMORY
2015-02-12 16:12:52,928 INFO reduce.InMemoryMapOutput (InMemoryMapOutput.java:shuffle(100)) - Read 200 bytes from map-output for attempt_local1666493537_0029_m_000000_0
2015-02-12 16:12:52,928 INFO reduce.MergeManagerImpl (MergeManagerImpl.java:closeInMemoryFile(313)) - closeInMemoryFile -> map-output of size: 200, inMemoryMapOutputs.size() -> 1, commitMemory -> 0, usedMemory ->200
2015-02-12 16:12:52,929 INFO reduce.EventFetcher (EventFetcher.java:run(76)) - EventFetcher is interrupted.. Returning
2015-02-12 16:12:52,929 INFO mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) - 1 / 1 copied.
2015-02-12 16:12:52,929 INFO reduce.MergeManagerImpl (MergeManagerImpl.java:finalMerge(673)) - finalMerge called with 1 in-memory map-outputs and 0 on-disk map-outputs
2015-02-12 16:12:52,930 INFO mapred.Merger (Merger.java:merge(606)) - Merging 1 sorted segments
2015-02-12 16:12:52,931 INFO mapred.Merger (Merger.java:merge(705)) - Down to the last merge-pass, with 1 segments left of total size: 191 bytes
2015-02-12 16:12:52,932 INFO reduce.MergeManagerImpl (MergeManagerImpl.java:finalMerge(750)) - Merged 1 segments, 200 bytes to disk to satisfy reduce memory limit
2015-02-12 16:12:52,932 INFO reduce.MergeManagerImpl (MergeManagerImpl.java:finalMerge(780)) - Merging 1 files, 204 bytes from disk
2015-02-12 16:12:52,933 INFO reduce.MergeManagerImpl (MergeManagerImpl.java:finalMerge(795)) - Merging 0 segments, 0 bytes from memory into reduce
2015-02-12 16:12:52,933 INFO mapred.Merger (Merger.java:merge(606)) - Merging 1 sorted segments
2015-02-12 16:12:52,933 INFO mapred.Merger (Merger.java:merge(705)) - Down to the last merge-pass, with 1 segments left of total size: 191 bytes
2015-02-12 16:12:52,933 INFO mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) - 1 / 1 copied.
2015-02-12 16:12:52,934 INFO hadoop.FlowReducer (FlowReducer.java:configure(78)) - cascading version: 2.6.3
2015-02-12 16:12:52,934 INFO hadoop.FlowReducer (FlowReducer.java:configure(79)) - child jvm opts: -Xmx200m
2015-02-12 16:12:52,937 INFO hadoop.FlowReducer (FlowReducer.java:configure(97)) - sourcing from: GroupBy(average)[by:[{1}:'char']]
2015-02-12 16:12:52,937 INFO hadoop.FlowReducer (FlowReducer.java:configure(100)) - sinking to: Hfs["TextDelimited[['char', 'average']]"]["/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/test/output/hadoop2-mr1/assemblyhelpersplatform/averagenull"]
2015-02-12 16:12:52,944 INFO mapred.Task (Task.java:done(1001)) - Task:attempt_local1666493537_0029_r_000000_0 is done. And is in the process of committing
2015-02-12 16:12:52,945 INFO mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) - 1 / 1 copied.
2015-02-12 16:12:52,945 INFO mapred.Task (Task.java:commit(1162)) - Task attempt_local1666493537_0029_r_000000_0 is allowed to commit now
2015-02-12 16:12:52,946 INFO output.FileOutputCommitter (FileOutputCommitter.java:commitTask(439)) - Saved output of task 'attempt_local1666493537_0029_r_000000_0' to file:/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/test/output/hadoop2-mr1/assemblyhelpersplatform/averagenull/_temporary/0/task_local1666493537_0029_r_000000
2015-02-12 16:12:52,946 INFO mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) - reduce > reduce
2015-02-12 16:12:52,947 INFO mapred.Task (Task.java:sendDone(1121)) - Task 'attempt_local1666493537_0029_r_000000_0' done.
2015-02-12 16:12:52,947 INFO mapred.LocalJobRunner (LocalJobRunner.java:run(325)) - Finishing task: attempt_local1666493537_0029_r_000000_0
2015-02-12 16:12:52,947 INFO mapred.LocalJobRunner (LocalJobRunner.java:runTasks(456)) - reduce task executor complete.
2015-02-12 16:12:52,957 INFO util.Hadoop18TapUtil (Hadoop18TapUtil.java:cleanTempPath(219)) - deleting temp path /Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/test/output/hadoop2-mr1/assemblyhelpersplatform/averagenull/_temporary
2015-02-12 16:12:52,982 INFO mapred.FileInputFormat (FileInputFormat.java:listStatus(247)) - Total input paths to process : 1
2015-02-12 16:12:53,005 INFO mapred.FileInputFormat (FileInputFormat.java:listStatus(247)) - Total input paths to process : 1
2015-02-12 16:12:53,006 INFO cascading.PlatformTestCase (PlatformTestCase.java:tearDown(209)) - copying to local /Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/test/output//hadoop2-mr1/assemblyhelpersplatform/averagenull
2015-02-12 16:12:53,017 INFO util.HadoopUtil (HadoopUtil.java:findMainClass(336)) - using default application jar, may cause class not found exceptions on the cluster
2015-02-12 16:12:53,017 INFO planner.HadoopPlanner (HadoopPlanner.java:initialize(225)) - using application jar: /Users/soberoi/.gradle/caches/modules-2/files-2.1/cascading/cascading-hadoop2-mr1/2.6.3/9ae016e2b8d246b9c1de9168e715d9f1b971aa4a/cascading-hadoop2-mr1-2.6.3.jar
2015-02-12 16:12:53,020 INFO Configuration.deprecation (Configuration.java:warnOnceIfDeprecated(1022)) - mapred.used.genericoptionsparser is deprecated. Instead, use mapreduce.client.genericoptionsparser.used
2015-02-12 16:12:53,022 INFO Configuration.deprecation (Configuration.java:warnOnceIfDeprecated(1022)) - mapred.job.tracker is deprecated. Instead, use mapreduce.jobtracker.address
2015-02-12 16:12:53,022 INFO Configuration.deprecation (Configuration.java:warnOnceIfDeprecated(1022)) - mapred.output.compress is deprecated. Instead, use mapreduce.output.fileoutputformat.compress
2015-02-12 16:12:53,025 INFO flow.Flow (BaseFlow.java:logInfo(1378)) - [coerce] starting
2015-02-12 16:12:53,026 INFO flow.Flow (BaseFlow.java:logInfo(1378)) - [coerce] source: Hfs["TextLine[['offset', 'line']->[ALL]]"]["/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/classes/test/data/lower.txt"]
2015-02-12 16:12:53,026 INFO flow.Flow (BaseFlow.java:logInfo(1378)) - [coerce] sink: Hfs["TextLine[['line']->['num', 'char']]"]["/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/test/output/hadoop2-mr1/assemblyhelpersplatform/coerce"]
2015-02-12 16:12:53,026 INFO flow.Flow (BaseFlow.java:logInfo(1378)) - [coerce] parallel execution is enabled: false
2015-02-12 16:12:53,026 INFO flow.Flow (BaseFlow.java:logInfo(1378)) - [coerce] starting jobs: 1
2015-02-12 16:12:53,026 INFO flow.Flow (BaseFlow.java:logInfo(1378)) - [coerce] allocating threads: 1
2015-02-12 16:12:53,026 INFO flow.FlowStep (BaseFlowStep.java:logInfo(834)) - [coerce] starting step: (1/1) ...blyhelpersplatform/coerce
2015-02-12 16:12:53,027 INFO jvm.JvmMetrics (JvmMetrics.java:init(71)) - Cannot initialize JVM Metrics with processName=JobTracker, sessionId= - already initialized
2015-02-12 16:12:53,028 INFO jvm.JvmMetrics (JvmMetrics.java:init(71)) - Cannot initialize JVM Metrics with processName=JobTracker, sessionId= - already initialized
2015-02-12 16:12:53,047 INFO mapred.FileInputFormat (FileInputFormat.java:listStatus(247)) - Total input paths to process : 1
2015-02-12 16:12:53,064 INFO mapreduce.JobSubmitter (JobSubmitter.java:submitJobInternal(492)) - number of splits:1
2015-02-12 16:12:53,075 INFO mapreduce.JobSubmitter (JobSubmitter.java:printTokens(575)) - Submitting tokens for job: job_local1609083427_0030
2015-02-12 16:12:53,084 WARN conf.Configuration (Configuration.java:loadProperty(2510)) - file:/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/tmp/cascading/staging/soberoi1609083427/.staging/job_local1609083427_0030/job.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.retry.interval; Ignoring.
2015-02-12 16:12:53,085 WARN conf.Configuration (Configuration.java:loadProperty(2510)) - file:/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/tmp/cascading/staging/soberoi1609083427/.staging/job_local1609083427_0030/job.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.attempts; Ignoring.
2015-02-12 16:12:53,118 WARN conf.Configuration (Configuration.java:loadProperty(2510)) - file:/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/test/tmp/mapred/local/localRunner/soberoi/job_local1609083427_0030/job_local1609083427_0030.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.retry.interval; Ignoring.
2015-02-12 16:12:53,120 WARN conf.Configuration (Configuration.java:loadProperty(2510)) - file:/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/test/tmp/mapred/local/localRunner/soberoi/job_local1609083427_0030/job_local1609083427_0030.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.attempts; Ignoring.
2015-02-12 16:12:53,120 INFO mapreduce.Job (Job.java:submit(1299)) - The url to track the job: http://localhost:8080/
2015-02-12 16:12:53,121 INFO flow.FlowStep (BaseFlowStep.java:logInfo(834)) - [coerce] submitted hadoop job: job_local1609083427_0030
2015-02-12 16:12:53,121 INFO mapred.LocalJobRunner (LocalJobRunner.java:createOutputCommitter(471)) - OutputCommitter set in config null
2015-02-12 16:12:53,121 INFO mapred.LocalJobRunner (LocalJobRunner.java:createOutputCommitter(489)) - OutputCommitter is org.apache.hadoop.mapred.FileOutputCommitter
2015-02-12 16:12:53,121 INFO flow.FlowStep (BaseFlowStep.java:logInfo(834)) - [coerce] tracking url: http://localhost:8080/
2015-02-12 16:12:53,123 INFO mapred.LocalJobRunner (LocalJobRunner.java:runTasks(448)) - Waiting for map tasks
2015-02-12 16:12:53,123 INFO mapred.LocalJobRunner (LocalJobRunner.java:run(224)) - Starting task: attempt_local1609083427_0030_m_000000_0
2015-02-12 16:12:53,124 INFO util.ProcfsBasedProcessTree (ProcfsBasedProcessTree.java:isAvailable(181)) - ProcfsBasedProcessTree currently is supported only on Linux.
2015-02-12 16:12:53,124 INFO mapred.Task (Task.java:initialize(587)) - Using ResourceCalculatorProcessTree : null
2015-02-12 16:12:53,125 INFO io.MultiInputSplit (MultiInputSplit.java:readFields(161)) - current split input path: file:/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/classes/test/data/lower.txt
2015-02-12 16:12:53,125 INFO mapred.MapTask (MapTask.java:updateJobWithSplit(482)) - Processing split: cascading.tap.hadoop.io.MultiInputSplit@4d15f20a
2015-02-12 16:12:53,126 INFO mapred.MapTask (MapTask.java:runOldMapper(436)) - numReduceTasks: 0
2015-02-12 16:12:53,134 INFO hadoop.FlowMapper (FlowMapper.java:configure(71)) - cascading version: 2.6.3
2015-02-12 16:12:53,134 INFO hadoop.FlowMapper (FlowMapper.java:configure(72)) - child jvm opts: -Xmx200m
2015-02-12 16:12:53,136 INFO hadoop.FlowMapper (FlowMapper.java:configure(87)) - sourcing from: Hfs["TextLine[['offset', 'line']->[ALL]]"]["/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/classes/test/data/lower.txt"]
2015-02-12 16:12:53,137 INFO hadoop.FlowMapper (FlowMapper.java:configure(90)) - sinking to: Hfs["TextLine[['line']->['num', 'char']]"]["/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/test/output/hadoop2-mr1/assemblyhelpersplatform/coerce"]
2015-02-12 16:12:53,139 INFO mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) -
2015-02-12 16:12:53,139 INFO mapred.Task (Task.java:done(1001)) - Task:attempt_local1609083427_0030_m_000000_0 is done. And is in the process of committing
2015-02-12 16:12:53,140 INFO mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) -
2015-02-12 16:12:53,140 INFO mapred.Task (Task.java:commit(1162)) - Task attempt_local1609083427_0030_m_000000_0 is allowed to commit now
2015-02-12 16:12:53,141 INFO output.FileOutputCommitter (FileOutputCommitter.java:commitTask(439)) - Saved output of task 'attempt_local1609083427_0030_m_000000_0' to file:/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/test/output/hadoop2-mr1/assemblyhelpersplatform/coerce/_temporary/0/task_local1609083427_0030_m_000000
2015-02-12 16:12:53,141 INFO mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) - file:/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/classes/test/data/lower.txt:0+19
2015-02-12 16:12:53,142 INFO mapred.Task (Task.java:sendDone(1121)) - Task 'attempt_local1609083427_0030_m_000000_0' done.
2015-02-12 16:12:53,142 INFO mapred.LocalJobRunner (LocalJobRunner.java:run(249)) - Finishing task: attempt_local1609083427_0030_m_000000_0
2015-02-12 16:12:53,142 INFO mapred.LocalJobRunner (LocalJobRunner.java:runTasks(456)) - map task executor complete.
2015-02-12 16:12:53,162 INFO util.Hadoop18TapUtil (Hadoop18TapUtil.java:cleanTempPath(219)) - deleting temp path /Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/test/output/hadoop2-mr1/assemblyhelpersplatform/coerce/_temporary
2015-02-12 16:12:53,189 INFO mapred.FileInputFormat (FileInputFormat.java:listStatus(247)) - Total input paths to process : 1
2015-02-12 16:12:53,190 INFO cascading.PlatformTestCase (PlatformTestCase.java:tearDown(209)) - copying to local /Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/test/output//hadoop2-mr1/assemblyhelpersplatform/coerce
2015-02-12 16:12:53,208 INFO util.HadoopUtil (HadoopUtil.java:findMainClass(336)) - using default application jar, may cause class not found exceptions on the cluster
2015-02-12 16:12:53,208 INFO planner.HadoopPlanner (HadoopPlanner.java:initialize(225)) - using application jar: /Users/soberoi/.gradle/caches/modules-2/files-2.1/cascading/cascading-hadoop2-mr1/2.6.3/9ae016e2b8d246b9c1de9168e715d9f1b971aa4a/cascading-hadoop2-mr1-2.6.3.jar
2015-02-12 16:12:53,213 INFO Configuration.deprecation (Configuration.java:warnOnceIfDeprecated(1022)) - mapred.used.genericoptionsparser is deprecated. Instead, use mapreduce.client.genericoptionsparser.used
2015-02-12 16:12:53,214 INFO Configuration.deprecation (Configuration.java:warnOnceIfDeprecated(1022)) - mapred.job.tracker is deprecated. Instead, use mapreduce.jobtracker.address
2015-02-12 16:12:53,215 INFO Configuration.deprecation (Configuration.java:warnOnceIfDeprecated(1022)) - mapred.output.compress is deprecated. Instead, use mapreduce.output.fileoutputformat.compress
2015-02-12 16:12:53,218 INFO flow.Flow (BaseFlow.java:logInfo(1378)) - [first] starting
2015-02-12 16:12:53,218 INFO flow.Flow (BaseFlow.java:logInfo(1378)) - [first] source: Hfs["TextDelimited[['num', 'lower', 'upper']]"]["/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/classes/test/data/lhs+rhs-cross-rev.txt"]
2015-02-12 16:12:53,218 INFO flow.Flow (BaseFlow.java:logInfo(1378)) - [first] sink: Hfs["TextDelimited[['num', 'lower', 'upper']]"]["/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/test/output/hadoop2-mr1/assemblyhelpersplatform/firstnfieldswithoutcomparator"]
2015-02-12 16:12:53,218 INFO flow.Flow (BaseFlow.java:logInfo(1378)) - [first] parallel execution is enabled: false
2015-02-12 16:12:53,218 INFO flow.Flow (BaseFlow.java:logInfo(1378)) - [first] starting jobs: 1
2015-02-12 16:12:53,218 INFO flow.Flow (BaseFlow.java:logInfo(1378)) - [first] allocating threads: 1
2015-02-12 16:12:53,219 INFO flow.FlowStep (BaseFlowStep.java:logInfo(834)) - [first] starting step: (1/1) ...tnfieldswithoutcomparator
2015-02-12 16:12:53,219 INFO jvm.JvmMetrics (JvmMetrics.java:init(71)) - Cannot initialize JVM Metrics with processName=JobTracker, sessionId= - already initialized
2015-02-12 16:12:53,220 INFO jvm.JvmMetrics (JvmMetrics.java:init(71)) - Cannot initialize JVM Metrics with processName=JobTracker, sessionId= - already initialized
2015-02-12 16:12:53,239 INFO mapred.FileInputFormat (FileInputFormat.java:listStatus(247)) - Total input paths to process : 1
2015-02-12 16:12:53,255 INFO mapreduce.JobSubmitter (JobSubmitter.java:submitJobInternal(492)) - number of splits:1
2015-02-12 16:12:53,267 INFO mapreduce.JobSubmitter (JobSubmitter.java:printTokens(575)) - Submitting tokens for job: job_local45421486_0031
2015-02-12 16:12:53,277 WARN conf.Configuration (Configuration.java:loadProperty(2510)) - file:/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/tmp/cascading/staging/soberoi45421486/.staging/job_local45421486_0031/job.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.retry.interval; Ignoring.
2015-02-12 16:12:53,278 WARN conf.Configuration (Configuration.java:loadProperty(2510)) - file:/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/tmp/cascading/staging/soberoi45421486/.staging/job_local45421486_0031/job.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.attempts; Ignoring.
2015-02-12 16:12:53,307 WARN conf.Configuration (Configuration.java:loadProperty(2510)) - file:/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/test/tmp/mapred/local/localRunner/soberoi/job_local45421486_0031/job_local45421486_0031.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.retry.interval; Ignoring.
2015-02-12 16:12:53,308 WARN conf.Configuration (Configuration.java:loadProperty(2510)) - file:/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/test/tmp/mapred/local/localRunner/soberoi/job_local45421486_0031/job_local45421486_0031.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.attempts; Ignoring.
2015-02-12 16:12:53,309 INFO mapreduce.Job (Job.java:submit(1299)) - The url to track the job: http://localhost:8080/
2015-02-12 16:12:53,314 INFO mapred.LocalJobRunner (LocalJobRunner.java:createOutputCommitter(471)) - OutputCommitter set in config null
2015-02-12 16:12:53,314 INFO mapred.LocalJobRunner (LocalJobRunner.java:createOutputCommitter(489)) - OutputCommitter is org.apache.hadoop.mapred.FileOutputCommitter
2015-02-12 16:12:53,314 INFO flow.FlowStep (BaseFlowStep.java:logInfo(834)) - [first] submitted hadoop job: job_local45421486_0031
2015-02-12 16:12:53,315 INFO flow.FlowStep (BaseFlowStep.java:logInfo(834)) - [first] tracking url: http://localhost:8080/
2015-02-12 16:12:53,316 INFO mapred.LocalJobRunner (LocalJobRunner.java:runTasks(448)) - Waiting for map tasks
2015-02-12 16:12:53,316 INFO mapred.LocalJobRunner (LocalJobRunner.java:run(224)) - Starting task: attempt_local45421486_0031_m_000000_0
2015-02-12 16:12:53,317 INFO util.ProcfsBasedProcessTree (ProcfsBasedProcessTree.java:isAvailable(181)) - ProcfsBasedProcessTree currently is supported only on Linux.
2015-02-12 16:12:53,318 INFO mapred.Task (Task.java:initialize(587)) - Using ResourceCalculatorProcessTree : null
2015-02-12 16:12:53,319 INFO io.MultiInputSplit (MultiInputSplit.java:readFields(161)) - current split input path: file:/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/classes/test/data/lhs+rhs-cross-rev.txt
2015-02-12 16:12:53,319 INFO mapred.MapTask (MapTask.java:updateJobWithSplit(482)) - Processing split: cascading.tap.hadoop.io.MultiInputSplit@2a3df98e
2015-02-12 16:12:53,319 INFO mapred.MapTask (MapTask.java:runOldMapper(436)) - numReduceTasks: 1
2015-02-12 16:12:53,331 INFO mapred.MapTask (MapTask.java:setEquator(1202)) - (EQUATOR) 0 kvi 26214396(104857584)
2015-02-12 16:12:53,331 INFO mapred.MapTask (MapTask.java:init(995)) - mapreduce.task.io.sort.mb: 100
2015-02-12 16:12:53,331 INFO mapred.MapTask (MapTask.java:init(996)) - soft limit at 83886080
2015-02-12 16:12:53,331 INFO mapred.MapTask (MapTask.java:init(997)) - bufstart = 0; bufvoid = 104857600
2015-02-12 16:12:53,331 INFO mapred.MapTask (MapTask.java:init(998)) - kvstart = 26214396; length = 6553600
2015-02-12 16:12:53,332 INFO mapred.MapTask (MapTask.java:createSortingCollector(402)) - Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer
2015-02-12 16:12:53,332 INFO hadoop.FlowMapper (FlowMapper.java:configure(71)) - cascading version: 2.6.3
2015-02-12 16:12:53,332 INFO hadoop.FlowMapper (FlowMapper.java:configure(72)) - child jvm opts: -Xmx200m
2015-02-12 16:12:53,335 INFO hadoop.FlowMapper (FlowMapper.java:configure(87)) - sourcing from: Hfs["TextDelimited[['num', 'lower', 'upper']]"]["/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/classes/test/data/lhs+rhs-cross-rev.txt"]
2015-02-12 16:12:53,335 INFO hadoop.FlowMapper (FlowMapper.java:configure(90)) - sinking to: GroupBy(first)[by:[{1}:'num']]
2015-02-12 16:12:53,336 INFO assembly.AggregateBy (AggregateBy.java:prepare(267)) - using threshold value: 2
2015-02-12 16:12:53,337 INFO assembly.AggregateBy (AggregateBy.java:removeEldestEntry(327)) - flushed keys num times: 1, with threshold: 2
2015-02-12 16:12:53,337 INFO assembly.AggregateBy (AggregateBy.java:removeEldestEntry(328)) - mem on flush (mb), free: 193, total: 513, max: 672
2015-02-12 16:12:53,337 INFO assembly.AggregateBy (AggregateBy.java:removeEldestEntry(333)) - total mem is 76% of max mem, to better utilize unused memory consider increasing current LRU threshold with system property "cascading.aggregateby.threshold"
2015-02-12 16:12:53,338 INFO assembly.AggregateBy (AggregateBy.java:removeEldestEntry(327)) - flushed keys num times: 3, with threshold: 2
2015-02-12 16:12:53,338 INFO assembly.AggregateBy (AggregateBy.java:removeEldestEntry(328)) - mem on flush (mb), free: 193, total: 513, max: 672
2015-02-12 16:12:53,338 INFO assembly.AggregateBy (AggregateBy.java:removeEldestEntry(333)) - total mem is 76% of max mem, to better utilize unused memory consider increasing current LRU threshold with system property "cascading.aggregateby.threshold"
2015-02-12 16:12:53,339 INFO mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) -
2015-02-12 16:12:53,339 INFO mapred.MapTask (MapTask.java:flush(1457)) - Starting flush of map output
2015-02-12 16:12:53,339 INFO mapred.MapTask (MapTask.java:flush(1475)) - Spilling map output
2015-02-12 16:12:53,340 INFO mapred.MapTask (MapTask.java:flush(1476)) - bufstart = 0; bufend = 105; bufvoid = 104857600
2015-02-12 16:12:53,340 INFO mapred.MapTask (MapTask.java:flush(1478)) - kvstart = 26214396(104857584); kvend = 26214380(104857520); length = 17/6553600
2015-02-12 16:12:53,341 INFO mapred.MapTask (MapTask.java:sortAndSpill(1660)) - Finished spill 0
2015-02-12 16:12:53,342 INFO mapred.Task (Task.java:done(1001)) - Task:attempt_local45421486_0031_m_000000_0 is done. And is in the process of committing
2015-02-12 16:12:53,343 INFO mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) - file:/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/classes/test/data/lhs+rhs-cross-rev.txt:0+221
2015-02-12 16:12:53,344 INFO mapred.Task (Task.java:sendDone(1121)) - Task 'attempt_local45421486_0031_m_000000_0' done.
2015-02-12 16:12:53,344 INFO mapred.LocalJobRunner (LocalJobRunner.java:run(249)) - Finishing task: attempt_local45421486_0031_m_000000_0
2015-02-12 16:12:53,344 INFO mapred.LocalJobRunner (LocalJobRunner.java:runTasks(456)) - map task executor complete.
2015-02-12 16:12:53,344 INFO mapred.LocalJobRunner (LocalJobRunner.java:runTasks(448)) - Waiting for reduce tasks
2015-02-12 16:12:53,344 INFO mapred.LocalJobRunner (LocalJobRunner.java:run(302)) - Starting task: attempt_local45421486_0031_r_000000_0
2015-02-12 16:12:53,345 INFO util.ProcfsBasedProcessTree (ProcfsBasedProcessTree.java:isAvailable(181)) - ProcfsBasedProcessTree currently is supported only on Linux.
2015-02-12 16:12:53,345 INFO mapred.Task (Task.java:initialize(587)) - Using ResourceCalculatorProcessTree : null
2015-02-12 16:12:53,345 INFO mapred.ReduceTask (ReduceTask.java:run(362)) - Using ShuffleConsumerPlugin: org.apache.hadoop.mapreduce.task.reduce.Shuffle@1ee3e282
2015-02-12 16:12:53,346 INFO reduce.MergeManagerImpl (MergeManagerImpl.java:<init>(195)) - MergerManager: memoryLimit=493250144, maxSingleShuffleLimit=123312536, mergeThreshold=325545120, ioSortFactor=10, memToMemMergeOutputsThreshold=10
2015-02-12 16:12:53,346 INFO reduce.EventFetcher (EventFetcher.java:run(61)) - attempt_local45421486_0031_r_000000_0 Thread started: EventFetcher for fetching Map Completion Events
2015-02-12 16:12:53,347 INFO reduce.LocalFetcher (LocalFetcher.java:copyMapOutput(141)) - localfetcher#24 about to shuffle output of map attempt_local45421486_0031_m_000000_0 decomp: 117 len: 121 to MEMORY
2015-02-12 16:12:53,347 INFO reduce.InMemoryMapOutput (InMemoryMapOutput.java:shuffle(100)) - Read 117 bytes from map-output for attempt_local45421486_0031_m_000000_0
2015-02-12 16:12:53,347 INFO reduce.MergeManagerImpl (MergeManagerImpl.java:closeInMemoryFile(313)) - closeInMemoryFile -> map-output of size: 117, inMemoryMapOutputs.size() -> 1, commitMemory -> 0, usedMemory ->117
2015-02-12 16:12:53,347 INFO reduce.EventFetcher (EventFetcher.java:run(76)) - EventFetcher is interrupted.. Returning
2015-02-12 16:12:53,348 INFO mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) - 1 / 1 copied.
2015-02-12 16:12:53,348 INFO reduce.MergeManagerImpl (MergeManagerImpl.java:finalMerge(673)) - finalMerge called with 1 in-memory map-outputs and 0 on-disk map-outputs
2015-02-12 16:12:53,349 INFO mapred.Merger (Merger.java:merge(606)) - Merging 1 sorted segments
2015-02-12 16:12:53,349 INFO mapred.Merger (Merger.java:merge(705)) - Down to the last merge-pass, with 1 segments left of total size: 108 bytes
2015-02-12 16:12:53,350 INFO reduce.MergeManagerImpl (MergeManagerImpl.java:finalMerge(750)) - Merged 1 segments, 117 bytes to disk to satisfy reduce memory limit
2015-02-12 16:12:53,350 INFO reduce.MergeManagerImpl (MergeManagerImpl.java:finalMerge(780)) - Merging 1 files, 121 bytes from disk
2015-02-12 16:12:53,350 INFO reduce.MergeManagerImpl (MergeManagerImpl.java:finalMerge(795)) - Merging 0 segments, 0 bytes from memory into reduce
2015-02-12 16:12:53,351 INFO mapred.Merger (Merger.java:merge(606)) - Merging 1 sorted segments
2015-02-12 16:12:53,351 INFO mapred.Merger (Merger.java:merge(705)) - Down to the last merge-pass, with 1 segments left of total size: 108 bytes
2015-02-12 16:12:53,351 INFO mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) - 1 / 1 copied.
2015-02-12 16:12:53,352 INFO hadoop.FlowReducer (FlowReducer.java:configure(78)) - cascading version: 2.6.3
2015-02-12 16:12:53,352 INFO hadoop.FlowReducer (FlowReducer.java:configure(79)) - child jvm opts: -Xmx200m
2015-02-12 16:12:53,355 INFO hadoop.FlowReducer (FlowReducer.java:configure(97)) - sourcing from: GroupBy(first)[by:[{1}:'num']]
2015-02-12 16:12:53,355 INFO hadoop.FlowReducer (FlowReducer.java:configure(100)) - sinking to: Hfs["TextDelimited[['num', 'lower', 'upper']]"]["/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/test/output/hadoop2-mr1/assemblyhelpersplatform/firstnfieldswithoutcomparator"]
2015-02-12 16:12:53,361 INFO mapred.Task (Task.java:done(1001)) - Task:attempt_local45421486_0031_r_000000_0 is done. And is in the process of committing
2015-02-12 16:12:53,362 INFO mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) - 1 / 1 copied.
2015-02-12 16:12:53,362 INFO mapred.Task (Task.java:commit(1162)) - Task attempt_local45421486_0031_r_000000_0 is allowed to commit now
2015-02-12 16:12:53,362 INFO output.FileOutputCommitter (FileOutputCommitter.java:commitTask(439)) - Saved output of task 'attempt_local45421486_0031_r_000000_0' to file:/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/test/output/hadoop2-mr1/assemblyhelpersplatform/firstnfieldswithoutcomparator/_temporary/0/task_local45421486_0031_r_000000
2015-02-12 16:12:53,363 INFO mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) - reduce > reduce
2015-02-12 16:12:53,363 INFO mapred.Task (Task.java:sendDone(1121)) - Task 'attempt_local45421486_0031_r_000000_0' done.
2015-02-12 16:12:53,363 INFO mapred.LocalJobRunner (LocalJobRunner.java:run(325)) - Finishing task: attempt_local45421486_0031_r_000000_0
2015-02-12 16:12:53,363 INFO mapred.LocalJobRunner (LocalJobRunner.java:runTasks(456)) - reduce task executor complete.
2015-02-12 16:12:53,374 INFO util.Hadoop18TapUtil (Hadoop18TapUtil.java:cleanTempPath(219)) - deleting temp path /Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/test/output/hadoop2-mr1/assemblyhelpersplatform/firstnfieldswithoutcomparator/_temporary
2015-02-12 16:12:53,394 INFO mapred.FileInputFormat (FileInputFormat.java:listStatus(247)) - Total input paths to process : 1
2015-02-12 16:12:53,395 INFO cascading.PlatformTestCase (PlatformTestCase.java:tearDown(209)) - copying to local /Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/test/output//hadoop2-mr1/assemblyhelpersplatform/firstnfieldswithoutcomparator
2015-02-12 16:12:53,407 INFO util.HadoopUtil (HadoopUtil.java:findMainClass(336)) - using default application jar, may cause class not found exceptions on the cluster
2015-02-12 16:12:53,408 INFO planner.HadoopPlanner (HadoopPlanner.java:initialize(225)) - using application jar: /Users/soberoi/.gradle/caches/modules-2/files-2.1/cascading/cascading-hadoop2-mr1/2.6.3/9ae016e2b8d246b9c1de9168e715d9f1b971aa4a/cascading-hadoop2-mr1-2.6.3.jar
2015-02-12 16:12:53,421 INFO Configuration.deprecation (Configuration.java:warnOnceIfDeprecated(1022)) - mapred.used.genericoptionsparser is deprecated. Instead, use mapreduce.client.genericoptionsparser.used
2015-02-12 16:12:53,422 INFO Configuration.deprecation (Configuration.java:warnOnceIfDeprecated(1022)) - mapred.job.tracker is deprecated. Instead, use mapreduce.jobtracker.address
2015-02-12 16:12:53,434 INFO Configuration.deprecation (Configuration.java:warnOnceIfDeprecated(1022)) - mapred.output.compress is deprecated. Instead, use mapreduce.output.fileoutputformat.compress
2015-02-12 16:12:53,439 INFO flow.Flow (BaseFlow.java:logInfo(1378)) - [count-lhs*source*count...] starting
2015-02-12 16:12:53,439 INFO flow.Flow (BaseFlow.java:logInfo(1378)) - [count-lhs*source*count...] source: Hfs["TextDelimited[['num', 'char']]"]["/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/classes/test/data/lhs.txt"]
2015-02-12 16:12:53,439 INFO flow.Flow (BaseFlow.java:logInfo(1378)) - [count-lhs*source*count...] sink: Hfs["TextDelimited[['char', 'count']]"]["/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/test/output/hadoop2-mr1/assemblyhelpersplatform/samesourcemergethreeway"]
2015-02-12 16:12:53,439 INFO flow.Flow (BaseFlow.java:logInfo(1378)) - [count-lhs*source*count...] parallel execution is enabled: false
2015-02-12 16:12:53,439 INFO flow.Flow (BaseFlow.java:logInfo(1378)) - [count-lhs*source*count...] starting jobs: 5
2015-02-12 16:12:53,439 INFO flow.Flow (BaseFlow.java:logInfo(1378)) - [count-lhs*source*count...] allocating threads: 1
2015-02-12 16:12:53,440 INFO flow.FlowStep (BaseFlowStep.java:logInfo(834)) - [count-lhs*source*count...] starting step: (1/5)
2015-02-12 16:12:53,440 INFO jvm.JvmMetrics (JvmMetrics.java:init(71)) - Cannot initialize JVM Metrics with processName=JobTracker, sessionId= - already initialized
2015-02-12 16:12:53,441 INFO jvm.JvmMetrics (JvmMetrics.java:init(71)) - Cannot initialize JVM Metrics with processName=JobTracker, sessionId= - already initialized
2015-02-12 16:12:53,458 INFO mapred.FileInputFormat (FileInputFormat.java:listStatus(247)) - Total input paths to process : 1
2015-02-12 16:12:53,472 INFO mapreduce.JobSubmitter (JobSubmitter.java:submitJobInternal(492)) - number of splits:1
2015-02-12 16:12:53,484 INFO mapreduce.JobSubmitter (JobSubmitter.java:printTokens(575)) - Submitting tokens for job: job_local2020268039_0032
2015-02-12 16:12:53,494 WARN conf.Configuration (Configuration.java:loadProperty(2510)) - file:/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/tmp/cascading/staging/soberoi2020268039/.staging/job_local2020268039_0032/job.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.retry.interval; Ignoring.
2015-02-12 16:12:53,495 WARN conf.Configuration (Configuration.java:loadProperty(2510)) - file:/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/tmp/cascading/staging/soberoi2020268039/.staging/job_local2020268039_0032/job.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.attempts; Ignoring.
2015-02-12 16:12:53,525 WARN conf.Configuration (Configuration.java:loadProperty(2510)) - file:/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/test/tmp/mapred/local/localRunner/soberoi/job_local2020268039_0032/job_local2020268039_0032.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.retry.interval; Ignoring.
2015-02-12 16:12:53,526 WARN conf.Configuration (Configuration.java:loadProperty(2510)) - file:/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/test/tmp/mapred/local/localRunner/soberoi/job_local2020268039_0032/job_local2020268039_0032.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.attempts; Ignoring.
2015-02-12 16:12:53,526 INFO mapreduce.Job (Job.java:submit(1299)) - The url to track the job: http://localhost:8080/
2015-02-12 16:12:53,527 INFO mapred.LocalJobRunner (LocalJobRunner.java:createOutputCommitter(471)) - OutputCommitter set in config null
2015-02-12 16:12:53,527 INFO flow.FlowStep (BaseFlowStep.java:logInfo(834)) - [count-lhs*source*count...] submitted hadoop job: job_local2020268039_0032
2015-02-12 16:12:53,527 INFO mapred.LocalJobRunner (LocalJobRunner.java:createOutputCommitter(489)) - OutputCommitter is org.apache.hadoop.mapred.FileOutputCommitter
2015-02-12 16:12:53,527 INFO flow.FlowStep (BaseFlowStep.java:logInfo(834)) - [count-lhs*source*count...] tracking url: http://localhost:8080/
2015-02-12 16:12:53,529 INFO mapred.LocalJobRunner (LocalJobRunner.java:runTasks(448)) - Waiting for map tasks
2015-02-12 16:12:53,529 INFO mapred.LocalJobRunner (LocalJobRunner.java:run(224)) - Starting task: attempt_local2020268039_0032_m_000000_0
2015-02-12 16:12:53,530 INFO util.ProcfsBasedProcessTree (ProcfsBasedProcessTree.java:isAvailable(181)) - ProcfsBasedProcessTree currently is supported only on Linux.
2015-02-12 16:12:53,530 INFO mapred.Task (Task.java:initialize(587)) - Using ResourceCalculatorProcessTree : null
2015-02-12 16:12:53,530 INFO io.MultiInputSplit (MultiInputSplit.java:readFields(161)) - current split input path: file:/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/classes/test/data/lhs.txt
2015-02-12 16:12:53,530 INFO mapred.MapTask (MapTask.java:updateJobWithSplit(482)) - Processing split: cascading.tap.hadoop.io.MultiInputSplit@6dfff19a
2015-02-12 16:12:53,531 INFO mapred.MapTask (MapTask.java:runOldMapper(436)) - numReduceTasks: 1
2015-02-12 16:12:53,546 INFO mapred.MapTask (MapTask.java:setEquator(1202)) - (EQUATOR) 0 kvi 26214396(104857584)
2015-02-12 16:12:53,546 INFO mapred.MapTask (MapTask.java:init(995)) - mapreduce.task.io.sort.mb: 100
2015-02-12 16:12:53,547 INFO mapred.MapTask (MapTask.java:init(996)) - soft limit at 83886080
2015-02-12 16:12:53,547 INFO mapred.MapTask (MapTask.java:init(997)) - bufstart = 0; bufvoid = 104857600
2015-02-12 16:12:53,547 INFO mapred.MapTask (MapTask.java:init(998)) - kvstart = 26214396; length = 6553600
2015-02-12 16:12:53,547 INFO mapred.MapTask (MapTask.java:createSortingCollector(402)) - Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer
2015-02-12 16:12:53,548 INFO hadoop.FlowMapper (FlowMapper.java:configure(71)) - cascading version: 2.6.3
2015-02-12 16:12:53,548 INFO hadoop.FlowMapper (FlowMapper.java:configure(72)) - child jvm opts: -Xmx200m
2015-02-12 16:12:53,551 INFO hadoop.FlowMapper (FlowMapper.java:configure(87)) - sourcing from: Hfs["TextDelimited[['num', 'char']]"]["/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/classes/test/data/lhs.txt"]
2015-02-12 16:12:53,551 INFO hadoop.FlowMapper (FlowMapper.java:configure(90)) - sinking to: GroupBy(source)[by:[{1}:'char']]
2015-02-12 16:12:53,551 INFO assembly.AggregateBy (AggregateBy.java:prepare(267)) - using threshold value: 2
2015-02-12 16:12:53,552 INFO assembly.AggregateBy (AggregateBy.java:removeEldestEntry(327)) - flushed keys num times: 1, with threshold: 2
2015-02-12 16:12:53,552 INFO assembly.AggregateBy (AggregateBy.java:removeEldestEntry(328)) - mem on flush (mb), free: 194, total: 513, max: 672
2015-02-12 16:12:53,552 INFO assembly.AggregateBy (AggregateBy.java:removeEldestEntry(333)) - total mem is 76% of max mem, to better utilize unused memory consider increasing current LRU threshold with system property "cascading.aggregateby.threshold"
2015-02-12 16:12:53,552 INFO assembly.AggregateBy (AggregateBy.java:removeEldestEntry(327)) - flushed keys num times: 3, with threshold: 2
2015-02-12 16:12:53,552 INFO assembly.AggregateBy (AggregateBy.java:removeEldestEntry(328)) - mem on flush (mb), free: 194, total: 513, max: 672
2015-02-12 16:12:53,552 INFO assembly.AggregateBy (AggregateBy.java:removeEldestEntry(333)) - total mem is 76% of max mem, to better utilize unused memory consider increasing current LRU threshold with system property "cascading.aggregateby.threshold"
2015-02-12 16:12:53,553 INFO assembly.AggregateBy (AggregateBy.java:removeEldestEntry(327)) - flushed keys num times: 5, with threshold: 2
2015-02-12 16:12:53,553 INFO assembly.AggregateBy (AggregateBy.java:removeEldestEntry(328)) - mem on flush (mb), free: 194, total: 513, max: 672
2015-02-12 16:12:53,553 INFO assembly.AggregateBy (AggregateBy.java:removeEldestEntry(333)) - total mem is 76% of max mem, to better utilize unused memory consider increasing current LRU threshold with system property "cascading.aggregateby.threshold"
2015-02-12 16:12:53,553 INFO assembly.AggregateBy (AggregateBy.java:removeEldestEntry(327)) - flushed keys num times: 7, with threshold: 2
2015-02-12 16:12:53,553 INFO assembly.AggregateBy (AggregateBy.java:removeEldestEntry(328)) - mem on flush (mb), free: 193, total: 513, max: 672
2015-02-12 16:12:53,553 INFO assembly.AggregateBy (AggregateBy.java:removeEldestEntry(333)) - total mem is 76% of max mem, to better utilize unused memory consider increasing current LRU threshold with system property "cascading.aggregateby.threshold"
2015-02-12 16:12:53,554 INFO mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) -
2015-02-12 16:12:53,554 INFO mapred.MapTask (MapTask.java:flush(1457)) - Starting flush of map output
2015-02-12 16:12:53,554 INFO mapred.MapTask (MapTask.java:flush(1475)) - Spilling map output
2015-02-12 16:12:53,554 INFO mapred.MapTask (MapTask.java:flush(1476)) - bufstart = 0; bufend = 99; bufvoid = 104857600
2015-02-12 16:12:53,555 INFO mapred.MapTask (MapTask.java:flush(1478)) - kvstart = 26214396(104857584); kvend = 26214364(104857456); length = 33/6553600
2015-02-12 16:12:53,556 INFO mapred.MapTask (MapTask.java:sortAndSpill(1660)) - Finished spill 0
2015-02-12 16:12:53,557 INFO mapred.Task (Task.java:done(1001)) - Task:attempt_local2020268039_0032_m_000000_0 is done. And is in the process of committing
2015-02-12 16:12:53,558 INFO mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) - file:/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/classes/test/data/lhs.txt:0+52
2015-02-12 16:12:53,558 INFO mapred.Task (Task.java:sendDone(1121)) - Task 'attempt_local2020268039_0032_m_000000_0' done.
2015-02-12 16:12:53,559 INFO mapred.LocalJobRunner (LocalJobRunner.java:run(249)) - Finishing task: attempt_local2020268039_0032_m_000000_0
2015-02-12 16:12:53,559 INFO mapred.LocalJobRunner (LocalJobRunner.java:runTasks(456)) - map task executor complete.
2015-02-12 16:12:53,559 INFO mapred.LocalJobRunner (LocalJobRunner.java:runTasks(448)) - Waiting for reduce tasks
2015-02-12 16:12:53,559 INFO mapred.LocalJobRunner (LocalJobRunner.java:run(302)) - Starting task: attempt_local2020268039_0032_r_000000_0
2015-02-12 16:12:53,560 INFO util.ProcfsBasedProcessTree (ProcfsBasedProcessTree.java:isAvailable(181)) - ProcfsBasedProcessTree currently is supported only on Linux.
2015-02-12 16:12:53,560 INFO mapred.Task (Task.java:initialize(587)) - Using ResourceCalculatorProcessTree : null
2015-02-12 16:12:53,560 INFO mapred.ReduceTask (ReduceTask.java:run(362)) - Using ShuffleConsumerPlugin: org.apache.hadoop.mapreduce.task.reduce.Shuffle@579ab801
2015-02-12 16:12:53,561 INFO reduce.MergeManagerImpl (MergeManagerImpl.java:<init>(195)) - MergerManager: memoryLimit=493250144, maxSingleShuffleLimit=123312536, mergeThreshold=325545120, ioSortFactor=10, memToMemMergeOutputsThreshold=10
2015-02-12 16:12:53,561 INFO reduce.EventFetcher (EventFetcher.java:run(61)) - attempt_local2020268039_0032_r_000000_0 Thread started: EventFetcher for fetching Map Completion Events
2015-02-12 16:12:53,562 INFO reduce.LocalFetcher (LocalFetcher.java:copyMapOutput(141)) - localfetcher#25 about to shuffle output of map attempt_local2020268039_0032_m_000000_0 decomp: 119 len: 123 to MEMORY
2015-02-12 16:12:53,563 INFO reduce.InMemoryMapOutput (InMemoryMapOutput.java:shuffle(100)) - Read 119 bytes from map-output for attempt_local2020268039_0032_m_000000_0
2015-02-12 16:12:53,563 INFO reduce.MergeManagerImpl (MergeManagerImpl.java:closeInMemoryFile(313)) - closeInMemoryFile -> map-output of size: 119, inMemoryMapOutputs.size() -> 1, commitMemory -> 0, usedMemory ->119
2015-02-12 16:12:53,563 INFO reduce.EventFetcher (EventFetcher.java:run(76)) - EventFetcher is interrupted.. Returning
2015-02-12 16:12:53,564 INFO mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) - 1 / 1 copied.
2015-02-12 16:12:53,564 INFO reduce.MergeManagerImpl (MergeManagerImpl.java:finalMerge(673)) - finalMerge called with 1 in-memory map-outputs and 0 on-disk map-outputs
2015-02-12 16:12:53,565 INFO mapred.Merger (Merger.java:merge(606)) - Merging 1 sorted segments
2015-02-12 16:12:53,565 INFO mapred.Merger (Merger.java:merge(705)) - Down to the last merge-pass, with 1 segments left of total size: 110 bytes
2015-02-12 16:12:53,566 INFO reduce.MergeManagerImpl (MergeManagerImpl.java:finalMerge(750)) - Merged 1 segments, 119 bytes to disk to satisfy reduce memory limit
2015-02-12 16:12:53,566 INFO reduce.MergeManagerImpl (MergeManagerImpl.java:finalMerge(780)) - Merging 1 files, 123 bytes from disk
2015-02-12 16:12:53,567 INFO reduce.MergeManagerImpl (MergeManagerImpl.java:finalMerge(795)) - Merging 0 segments, 0 bytes from memory into reduce
2015-02-12 16:12:53,567 INFO mapred.Merger (Merger.java:merge(606)) - Merging 1 sorted segments
2015-02-12 16:12:53,567 INFO mapred.Merger (Merger.java:merge(705)) - Down to the last merge-pass, with 1 segments left of total size: 110 bytes
2015-02-12 16:12:53,567 INFO mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) - 1 / 1 copied.
2015-02-12 16:12:53,568 INFO hadoop.FlowReducer (FlowReducer.java:configure(78)) - cascading version: 2.6.3
2015-02-12 16:12:53,568 INFO hadoop.FlowReducer (FlowReducer.java:configure(79)) - child jvm opts: -Xmx200m
2015-02-12 16:12:53,571 INFO hadoop.FlowReducer (FlowReducer.java:configure(97)) - sourcing from: GroupBy(source)[by:[{1}:'char']]
2015-02-12 16:12:53,571 INFO hadoop.FlowReducer (FlowReducer.java:configure(100)) - sinking to: TempHfs["SequenceFile[['char', 'count']]"][3798598062/source/]
2015-02-12 16:12:53,577 INFO mapred.Task (Task.java:done(1001)) - Task:attempt_local2020268039_0032_r_000000_0 is done. And is in the process of committing
2015-02-12 16:12:53,578 INFO mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) - 1 / 1 copied.
2015-02-12 16:12:53,578 INFO mapred.Task (Task.java:commit(1162)) - Task attempt_local2020268039_0032_r_000000_0 is allowed to commit now
2015-02-12 16:12:53,579 INFO output.FileOutputCommitter (FileOutputCommitter.java:commitTask(439)) - Saved output of task 'attempt_local2020268039_0032_r_000000_0' to file:/tmp/hadoop-soberoi/3798598062_source_BF20C2B770DC46CE8090B9EEF941B213/_temporary/0/task_local2020268039_0032_r_000000
2015-02-12 16:12:53,579 INFO mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) - reduce > reduce
2015-02-12 16:12:53,580 INFO mapred.Task (Task.java:sendDone(1121)) - Task 'attempt_local2020268039_0032_r_000000_0' done.
2015-02-12 16:12:53,580 INFO mapred.LocalJobRunner (LocalJobRunner.java:run(325)) - Finishing task: attempt_local2020268039_0032_r_000000_0
2015-02-12 16:12:53,580 INFO mapred.LocalJobRunner (LocalJobRunner.java:runTasks(456)) - reduce task executor complete.
2015-02-12 16:12:53,599 INFO flow.FlowStep (BaseFlowStep.java:logInfo(834)) - [count-lhs*source*count...] starting step: (3/5)
2015-02-12 16:12:53,600 INFO jvm.JvmMetrics (JvmMetrics.java:init(71)) - Cannot initialize JVM Metrics with processName=JobTracker, sessionId= - already initialized
2015-02-12 16:12:53,601 INFO jvm.JvmMetrics (JvmMetrics.java:init(71)) - Cannot initialize JVM Metrics with processName=JobTracker, sessionId= - already initialized
2015-02-12 16:12:53,631 INFO mapred.FileInputFormat (FileInputFormat.java:listStatus(247)) - Total input paths to process : 1
2015-02-12 16:12:53,646 INFO mapreduce.JobSubmitter (JobSubmitter.java:submitJobInternal(492)) - number of splits:1
2015-02-12 16:12:53,655 INFO Configuration.deprecation (Configuration.java:warnOnceIfDeprecated(1022)) - mapred.used.genericoptionsparser is deprecated. Instead, use mapreduce.client.genericoptionsparser.used
2015-02-12 16:12:53,655 INFO Configuration.deprecation (Configuration.java:warnOnceIfDeprecated(1022)) - mapred.job.tracker is deprecated. Instead, use mapreduce.jobtracker.address
2015-02-12 16:12:53,658 INFO mapreduce.JobSubmitter (JobSubmitter.java:printTokens(575)) - Submitting tokens for job: job_local1223674493_0033
2015-02-12 16:12:53,667 WARN conf.Configuration (Configuration.java:loadProperty(2510)) - file:/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/tmp/cascading/staging/soberoi1223674493/.staging/job_local1223674493_0033/job.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.retry.interval; Ignoring.
2015-02-12 16:12:53,668 WARN conf.Configuration (Configuration.java:loadProperty(2510)) - file:/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/tmp/cascading/staging/soberoi1223674493/.staging/job_local1223674493_0033/job.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.attempts; Ignoring.
2015-02-12 16:12:53,698 WARN conf.Configuration (Configuration.java:loadProperty(2510)) - file:/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/test/tmp/mapred/local/localRunner/soberoi/job_local1223674493_0033/job_local1223674493_0033.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.retry.interval; Ignoring.
2015-02-12 16:12:53,699 WARN conf.Configuration (Configuration.java:loadProperty(2510)) - file:/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/test/tmp/mapred/local/localRunner/soberoi/job_local1223674493_0033/job_local1223674493_0033.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.attempts; Ignoring.
2015-02-12 16:12:53,700 INFO mapreduce.Job (Job.java:submit(1299)) - The url to track the job: http://localhost:8080/
2015-02-12 16:12:53,700 INFO mapred.LocalJobRunner (LocalJobRunner.java:createOutputCommitter(471)) - OutputCommitter set in config null
2015-02-12 16:12:53,700 INFO flow.FlowStep (BaseFlowStep.java:logInfo(834)) - [count-lhs*source*count...] submitted hadoop job: job_local1223674493_0033
2015-02-12 16:12:53,700 INFO mapred.LocalJobRunner (LocalJobRunner.java:createOutputCommitter(489)) - OutputCommitter is org.apache.hadoop.mapred.FileOutputCommitter
2015-02-12 16:12:53,700 INFO flow.FlowStep (BaseFlowStep.java:logInfo(834)) - [count-lhs*source*count...] tracking url: http://localhost:8080/
2015-02-12 16:12:53,701 INFO mapred.LocalJobRunner (LocalJobRunner.java:runTasks(448)) - Waiting for map tasks
2015-02-12 16:12:53,702 INFO mapred.LocalJobRunner (LocalJobRunner.java:run(224)) - Starting task: attempt_local1223674493_0033_m_000000_0
2015-02-12 16:12:53,702 INFO util.ProcfsBasedProcessTree (ProcfsBasedProcessTree.java:isAvailable(181)) - ProcfsBasedProcessTree currently is supported only on Linux.
2015-02-12 16:12:53,703 INFO mapred.Task (Task.java:initialize(587)) - Using ResourceCalculatorProcessTree : null
2015-02-12 16:12:53,703 INFO io.MultiInputSplit (MultiInputSplit.java:readFields(161)) - current split input path: file:/tmp/hadoop-soberoi/3798598062_source_BF20C2B770DC46CE8090B9EEF941B213/part-00000
2015-02-12 16:12:53,703 INFO mapred.MapTask (MapTask.java:updateJobWithSplit(482)) - Processing split: cascading.tap.hadoop.io.MultiInputSplit@100a494b
2015-02-12 16:12:53,704 INFO mapred.MapTask (MapTask.java:runOldMapper(436)) - numReduceTasks: 1
2015-02-12 16:12:53,720 INFO mapred.MapTask (MapTask.java:setEquator(1202)) - (EQUATOR) 0 kvi 26214396(104857584)
2015-02-12 16:12:53,720 INFO mapred.MapTask (MapTask.java:init(995)) - mapreduce.task.io.sort.mb: 100
2015-02-12 16:12:53,720 INFO mapred.MapTask (MapTask.java:init(996)) - soft limit at 83886080
2015-02-12 16:12:53,720 INFO mapred.MapTask (MapTask.java:init(997)) - bufstart = 0; bufvoid = 104857600
2015-02-12 16:12:53,720 INFO mapred.MapTask (MapTask.java:init(998)) - kvstart = 26214396; length = 6553600
2015-02-12 16:12:53,721 INFO mapred.MapTask (MapTask.java:createSortingCollector(402)) - Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer
2015-02-12 16:12:53,721 INFO hadoop.FlowMapper (FlowMapper.java:configure(71)) - cascading version: 2.6.3
2015-02-12 16:12:53,722 INFO hadoop.FlowMapper (FlowMapper.java:configure(72)) - child jvm opts: -Xmx200m
2015-02-12 16:12:53,725 INFO hadoop.FlowMapper (FlowMapper.java:configure(87)) - sourcing from: TempHfs["SequenceFile[['char', 'count']]"][3798598062/source/]
2015-02-12 16:12:53,725 INFO hadoop.FlowMapper (FlowMapper.java:configure(90)) - sinking to: GroupBy(count-lhs)[by:[{1}:'char']]
2015-02-12 16:12:53,725 INFO assembly.AggregateBy (AggregateBy.java:prepare(267)) - using threshold value: 2
2015-02-12 16:12:53,726 INFO assembly.AggregateBy (AggregateBy.java:removeEldestEntry(327)) - flushed keys num times: 1, with threshold: 2
2015-02-12 16:12:53,726 INFO assembly.AggregateBy (AggregateBy.java:removeEldestEntry(328)) - mem on flush (mb), free: 195, total: 514, max: 672
2015-02-12 16:12:53,726 INFO assembly.AggregateBy (AggregateBy.java:removeEldestEntry(333)) - total mem is 76% of max mem, to better utilize unused memory consider increasing current LRU threshold with system property "cascading.aggregateby.threshold"
2015-02-12 16:12:53,726 INFO assembly.AggregateBy (AggregateBy.java:removeEldestEntry(327)) - flushed keys num times: 3, with threshold: 2
2015-02-12 16:12:53,726 INFO assembly.AggregateBy (AggregateBy.java:removeEldestEntry(328)) - mem on flush (mb), free: 195, total: 514, max: 672
2015-02-12 16:12:53,727 INFO assembly.AggregateBy (AggregateBy.java:removeEldestEntry(333)) - total mem is 76% of max mem, to better utilize unused memory consider increasing current LRU threshold with system property "cascading.aggregateby.threshold"
2015-02-12 16:12:53,727 INFO mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) -
2015-02-12 16:12:53,727 INFO mapred.MapTask (MapTask.java:flush(1457)) - Starting flush of map output
2015-02-12 16:12:53,727 INFO mapred.MapTask (MapTask.java:flush(1475)) - Spilling map output
2015-02-12 16:12:53,727 INFO mapred.MapTask (MapTask.java:flush(1476)) - bufstart = 0; bufend = 55; bufvoid = 104857600
2015-02-12 16:12:53,728 INFO mapred.MapTask (MapTask.java:flush(1478)) - kvstart = 26214396(104857584); kvend = 26214380(104857520); length = 17/6553600
2015-02-12 16:12:53,729 INFO mapred.MapTask (MapTask.java:sortAndSpill(1660)) - Finished spill 0
2015-02-12 16:12:53,730 INFO mapred.Task (Task.java:done(1001)) - Task:attempt_local1223674493_0033_m_000000_0 is done. And is in the process of committing
2015-02-12 16:12:53,731 INFO mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) - file:/tmp/hadoop-soberoi/3798598062_source_BF20C2B770DC46CE8090B9EEF941B213/part-00000:0+160
2015-02-12 16:12:53,731 INFO mapred.Task (Task.java:sendDone(1121)) - Task 'attempt_local1223674493_0033_m_000000_0' done.
2015-02-12 16:12:53,731 INFO mapred.LocalJobRunner (LocalJobRunner.java:run(249)) - Finishing task: attempt_local1223674493_0033_m_000000_0
2015-02-12 16:12:53,731 INFO mapred.LocalJobRunner (LocalJobRunner.java:runTasks(456)) - map task executor complete.
2015-02-12 16:12:53,732 INFO mapred.LocalJobRunner (LocalJobRunner.java:runTasks(448)) - Waiting for reduce tasks
2015-02-12 16:12:53,732 INFO mapred.LocalJobRunner (LocalJobRunner.java:run(302)) - Starting task: attempt_local1223674493_0033_r_000000_0
2015-02-12 16:12:53,733 INFO util.ProcfsBasedProcessTree (ProcfsBasedProcessTree.java:isAvailable(181)) - ProcfsBasedProcessTree currently is supported only on Linux.
2015-02-12 16:12:53,733 INFO mapred.Task (Task.java:initialize(587)) - Using ResourceCalculatorProcessTree : null
2015-02-12 16:12:53,733 INFO mapred.ReduceTask (ReduceTask.java:run(362)) - Using ShuffleConsumerPlugin: org.apache.hadoop.mapreduce.task.reduce.Shuffle@32f2e8ca
2015-02-12 16:12:53,733 INFO reduce.MergeManagerImpl (MergeManagerImpl.java:<init>(195)) - MergerManager: memoryLimit=493250144, maxSingleShuffleLimit=123312536, mergeThreshold=325545120, ioSortFactor=10, memToMemMergeOutputsThreshold=10
2015-02-12 16:12:53,734 INFO reduce.EventFetcher (EventFetcher.java:run(61)) - attempt_local1223674493_0033_r_000000_0 Thread started: EventFetcher for fetching Map Completion Events
2015-02-12 16:12:53,735 INFO reduce.LocalFetcher (LocalFetcher.java:copyMapOutput(141)) - localfetcher#26 about to shuffle output of map attempt_local1223674493_0033_m_000000_0 decomp: 67 len: 71 to MEMORY
2015-02-12 16:12:53,735 INFO reduce.InMemoryMapOutput (InMemoryMapOutput.java:shuffle(100)) - Read 67 bytes from map-output for attempt_local1223674493_0033_m_000000_0
2015-02-12 16:12:53,735 INFO reduce.MergeManagerImpl (MergeManagerImpl.java:closeInMemoryFile(313)) - closeInMemoryFile -> map-output of size: 67, inMemoryMapOutputs.size() -> 1, commitMemory -> 0, usedMemory ->67
2015-02-12 16:12:53,736 INFO reduce.EventFetcher (EventFetcher.java:run(76)) - EventFetcher is interrupted.. Returning
2015-02-12 16:12:53,736 INFO mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) - 1 / 1 copied.
2015-02-12 16:12:53,736 INFO reduce.MergeManagerImpl (MergeManagerImpl.java:finalMerge(673)) - finalMerge called with 1 in-memory map-outputs and 0 on-disk map-outputs
2015-02-12 16:12:53,737 INFO mapred.Merger (Merger.java:merge(606)) - Merging 1 sorted segments
2015-02-12 16:12:53,738 INFO mapred.Merger (Merger.java:merge(705)) - Down to the last merge-pass, with 1 segments left of total size: 58 bytes
2015-02-12 16:12:53,739 INFO reduce.MergeManagerImpl (MergeManagerImpl.java:finalMerge(750)) - Merged 1 segments, 67 bytes to disk to satisfy reduce memory limit
2015-02-12 16:12:53,739 INFO reduce.MergeManagerImpl (MergeManagerImpl.java:finalMerge(780)) - Merging 1 files, 71 bytes from disk
2015-02-12 16:12:53,739 INFO reduce.MergeManagerImpl (MergeManagerImpl.java:finalMerge(795)) - Merging 0 segments, 0 bytes from memory into reduce
2015-02-12 16:12:53,739 INFO mapred.Merger (Merger.java:merge(606)) - Merging 1 sorted segments
2015-02-12 16:12:53,740 INFO mapred.Merger (Merger.java:merge(705)) - Down to the last merge-pass, with 1 segments left of total size: 58 bytes
2015-02-12 16:12:53,740 INFO mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) - 1 / 1 copied.
2015-02-12 16:12:53,740 INFO hadoop.FlowReducer (FlowReducer.java:configure(78)) - cascading version: 2.6.3
2015-02-12 16:12:53,740 INFO hadoop.FlowReducer (FlowReducer.java:configure(79)) - child jvm opts: -Xmx200m
2015-02-12 16:12:53,744 INFO hadoop.FlowReducer (FlowReducer.java:configure(97)) - sourcing from: GroupBy(count-lhs)[by:[{1}:'char']]
2015-02-12 16:12:53,744 INFO hadoop.FlowReducer (FlowReducer.java:configure(100)) - sinking to: TempHfs["SequenceFile[['char', 'count']]"][6668818197/count-lhs/]
2015-02-12 16:12:53,751 INFO mapred.Task (Task.java:done(1001)) - Task:attempt_local1223674493_0033_r_000000_0 is done. And is in the process of committing
2015-02-12 16:12:53,752 INFO mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) - 1 / 1 copied.
2015-02-12 16:12:53,753 INFO mapred.Task (Task.java:commit(1162)) - Task attempt_local1223674493_0033_r_000000_0 is allowed to commit now
2015-02-12 16:12:53,753 INFO output.FileOutputCommitter (FileOutputCommitter.java:commitTask(439)) - Saved output of task 'attempt_local1223674493_0033_r_000000_0' to file:/tmp/hadoop-soberoi/6668818197_count_lhs_E0299A91C9AD4B61A1E657A2086B2F4E/_temporary/0/task_local1223674493_0033_r_000000
2015-02-12 16:12:53,754 INFO mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) - reduce > reduce
2015-02-12 16:12:53,754 INFO mapred.Task (Task.java:sendDone(1121)) - Task 'attempt_local1223674493_0033_r_000000_0' done.
2015-02-12 16:12:53,754 INFO mapred.LocalJobRunner (LocalJobRunner.java:run(325)) - Finishing task: attempt_local1223674493_0033_r_000000_0
2015-02-12 16:12:53,755 INFO mapred.LocalJobRunner (LocalJobRunner.java:runTasks(456)) - reduce task executor complete.
2015-02-12 16:12:53,773 INFO flow.FlowStep (BaseFlowStep.java:logInfo(834)) - [count-lhs*source*count...] starting step: (4/5)
2015-02-12 16:12:53,773 INFO jvm.JvmMetrics (JvmMetrics.java:init(71)) - Cannot initialize JVM Metrics with processName=JobTracker, sessionId= - already initialized
2015-02-12 16:12:53,774 INFO jvm.JvmMetrics (JvmMetrics.java:init(71)) - Cannot initialize JVM Metrics with processName=JobTracker, sessionId= - already initialized
2015-02-12 16:12:53,803 INFO mapred.FileInputFormat (FileInputFormat.java:listStatus(247)) - Total input paths to process : 1
2015-02-12 16:12:53,818 INFO mapreduce.JobSubmitter (JobSubmitter.java:submitJobInternal(492)) - number of splits:1
2015-02-12 16:12:53,827 INFO Configuration.deprecation (Configuration.java:warnOnceIfDeprecated(1022)) - mapred.used.genericoptionsparser is deprecated. Instead, use mapreduce.client.genericoptionsparser.used
2015-02-12 16:12:53,827 INFO Configuration.deprecation (Configuration.java:warnOnceIfDeprecated(1022)) - mapred.job.tracker is deprecated. Instead, use mapreduce.jobtracker.address
2015-02-12 16:12:53,831 INFO mapreduce.JobSubmitter (JobSubmitter.java:printTokens(575)) - Submitting tokens for job: job_local502182601_0034
2015-02-12 16:12:53,841 WARN conf.Configuration (Configuration.java:loadProperty(2510)) - file:/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/tmp/cascading/staging/soberoi502182601/.staging/job_local502182601_0034/job.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.retry.interval; Ignoring.
2015-02-12 16:12:53,842 WARN conf.Configuration (Configuration.java:loadProperty(2510)) - file:/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/tmp/cascading/staging/soberoi502182601/.staging/job_local502182601_0034/job.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.attempts; Ignoring.
2015-02-12 16:12:53,870 WARN conf.Configuration (Configuration.java:loadProperty(2510)) - file:/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/test/tmp/mapred/local/localRunner/soberoi/job_local502182601_0034/job_local502182601_0034.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.retry.interval; Ignoring.
2015-02-12 16:12:53,871 WARN conf.Configuration (Configuration.java:loadProperty(2510)) - file:/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/test/tmp/mapred/local/localRunner/soberoi/job_local502182601_0034/job_local502182601_0034.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.attempts; Ignoring.
2015-02-12 16:12:53,872 INFO mapreduce.Job (Job.java:submit(1299)) - The url to track the job: http://localhost:8080/
2015-02-12 16:12:53,872 INFO mapred.LocalJobRunner (LocalJobRunner.java:createOutputCommitter(471)) - OutputCommitter set in config null
2015-02-12 16:12:53,872 INFO flow.FlowStep (BaseFlowStep.java:logInfo(834)) - [count-lhs*source*count...] submitted hadoop job: job_local502182601_0034
2015-02-12 16:12:53,872 INFO mapred.LocalJobRunner (LocalJobRunner.java:createOutputCommitter(489)) - OutputCommitter is org.apache.hadoop.mapred.FileOutputCommitter
2015-02-12 16:12:53,872 INFO flow.FlowStep (BaseFlowStep.java:logInfo(834)) - [count-lhs*source*count...] tracking url: http://localhost:8080/
2015-02-12 16:12:53,874 INFO mapred.LocalJobRunner (LocalJobRunner.java:runTasks(448)) - Waiting for map tasks
2015-02-12 16:12:53,874 INFO mapred.LocalJobRunner (LocalJobRunner.java:run(224)) - Starting task: attempt_local502182601_0034_m_000000_0
2015-02-12 16:12:53,875 INFO util.ProcfsBasedProcessTree (ProcfsBasedProcessTree.java:isAvailable(181)) - ProcfsBasedProcessTree currently is supported only on Linux.
2015-02-12 16:12:53,875 INFO mapred.Task (Task.java:initialize(587)) - Using ResourceCalculatorProcessTree : null
2015-02-12 16:12:53,875 INFO io.MultiInputSplit (MultiInputSplit.java:readFields(161)) - current split input path: file:/tmp/hadoop-soberoi/3798598062_source_BF20C2B770DC46CE8090B9EEF941B213/part-00000
2015-02-12 16:12:53,876 INFO mapred.MapTask (MapTask.java:updateJobWithSplit(482)) - Processing split: cascading.tap.hadoop.io.MultiInputSplit@1854dfcf
2015-02-12 16:12:53,876 INFO mapred.MapTask (MapTask.java:runOldMapper(436)) - numReduceTasks: 1
2015-02-12 16:12:53,892 INFO mapred.MapTask (MapTask.java:setEquator(1202)) - (EQUATOR) 0 kvi 26214396(104857584)
2015-02-12 16:12:53,892 INFO mapred.MapTask (MapTask.java:init(995)) - mapreduce.task.io.sort.mb: 100
2015-02-12 16:12:53,892 INFO mapred.MapTask (MapTask.java:init(996)) - soft limit at 83886080
2015-02-12 16:12:53,892 INFO mapred.MapTask (MapTask.java:init(997)) - bufstart = 0; bufvoid = 104857600
2015-02-12 16:12:53,892 INFO mapred.MapTask (MapTask.java:init(998)) - kvstart = 26214396; length = 6553600
2015-02-12 16:12:53,893 INFO mapred.MapTask (MapTask.java:createSortingCollector(402)) - Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer
2015-02-12 16:12:53,893 INFO hadoop.FlowMapper (FlowMapper.java:configure(71)) - cascading version: 2.6.3
2015-02-12 16:12:53,893 INFO hadoop.FlowMapper (FlowMapper.java:configure(72)) - child jvm opts: -Xmx200m
2015-02-12 16:12:53,896 INFO hadoop.FlowMapper (FlowMapper.java:configure(87)) - sourcing from: TempHfs["SequenceFile[['char', 'count']]"][3798598062/source/]
2015-02-12 16:12:53,896 INFO hadoop.FlowMapper (FlowMapper.java:configure(90)) - sinking to: GroupBy(count-rhs)[by:[{1}:'char']]
2015-02-12 16:12:53,897 INFO assembly.AggregateBy (AggregateBy.java:prepare(267)) - using threshold value: 2
2015-02-12 16:12:53,898 INFO assembly.AggregateBy (AggregateBy.java:removeEldestEntry(327)) - flushed keys num times: 1, with threshold: 2
2015-02-12 16:12:53,898 INFO assembly.AggregateBy (AggregateBy.java:removeEldestEntry(328)) - mem on flush (mb), free: 195, total: 514, max: 672
2015-02-12 16:12:53,898 INFO assembly.AggregateBy (AggregateBy.java:removeEldestEntry(333)) - total mem is 76% of max mem, to better utilize unused memory consider increasing current LRU threshold with system property "cascading.aggregateby.threshold"
2015-02-12 16:12:53,898 INFO assembly.AggregateBy (AggregateBy.java:removeEldestEntry(327)) - flushed keys num times: 3, with threshold: 2
2015-02-12 16:12:53,898 INFO assembly.AggregateBy (AggregateBy.java:removeEldestEntry(328)) - mem on flush (mb), free: 195, total: 514, max: 672
2015-02-12 16:12:53,898 INFO assembly.AggregateBy (AggregateBy.java:removeEldestEntry(333)) - total mem is 76% of max mem, to better utilize unused memory consider increasing current LRU threshold with system property "cascading.aggregateby.threshold"
2015-02-12 16:12:53,899 INFO mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) -
2015-02-12 16:12:53,899 INFO mapred.MapTask (MapTask.java:flush(1457)) - Starting flush of map output
2015-02-12 16:12:53,899 INFO mapred.MapTask (MapTask.java:flush(1475)) - Spilling map output
2015-02-12 16:12:53,899 INFO mapred.MapTask (MapTask.java:flush(1476)) - bufstart = 0; bufend = 55; bufvoid = 104857600
2015-02-12 16:12:53,899 INFO mapred.MapTask (MapTask.java:flush(1478)) - kvstart = 26214396(104857584); kvend = 26214380(104857520); length = 17/6553600
2015-02-12 16:12:53,901 INFO mapred.MapTask (MapTask.java:sortAndSpill(1660)) - Finished spill 0
2015-02-12 16:12:53,902 INFO mapred.Task (Task.java:done(1001)) - Task:attempt_local502182601_0034_m_000000_0 is done. And is in the process of committing
2015-02-12 16:12:53,903 INFO mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) - file:/tmp/hadoop-soberoi/3798598062_source_BF20C2B770DC46CE8090B9EEF941B213/part-00000:0+160
2015-02-12 16:12:53,903 INFO mapred.Task (Task.java:sendDone(1121)) - Task 'attempt_local502182601_0034_m_000000_0' done.
2015-02-12 16:12:53,903 INFO mapred.LocalJobRunner (LocalJobRunner.java:run(249)) - Finishing task: attempt_local502182601_0034_m_000000_0
2015-02-12 16:12:53,903 INFO mapred.LocalJobRunner (LocalJobRunner.java:runTasks(456)) - map task executor complete.
2015-02-12 16:12:53,904 INFO mapred.LocalJobRunner (LocalJobRunner.java:runTasks(448)) - Waiting for reduce tasks
2015-02-12 16:12:53,904 INFO mapred.LocalJobRunner (LocalJobRunner.java:run(302)) - Starting task: attempt_local502182601_0034_r_000000_0
2015-02-12 16:12:53,905 INFO util.ProcfsBasedProcessTree (ProcfsBasedProcessTree.java:isAvailable(181)) - ProcfsBasedProcessTree currently is supported only on Linux.
2015-02-12 16:12:53,905 INFO mapred.Task (Task.java:initialize(587)) - Using ResourceCalculatorProcessTree : null
2015-02-12 16:12:53,905 INFO mapred.ReduceTask (ReduceTask.java:run(362)) - Using ShuffleConsumerPlugin: org.apache.hadoop.mapreduce.task.reduce.Shuffle@784c0b8
2015-02-12 16:12:53,905 INFO reduce.MergeManagerImpl (MergeManagerImpl.java:<init>(195)) - MergerManager: memoryLimit=493250144, maxSingleShuffleLimit=123312536, mergeThreshold=325545120, ioSortFactor=10, memToMemMergeOutputsThreshold=10
2015-02-12 16:12:53,906 INFO reduce.EventFetcher (EventFetcher.java:run(61)) - attempt_local502182601_0034_r_000000_0 Thread started: EventFetcher for fetching Map Completion Events
2015-02-12 16:12:53,906 INFO reduce.LocalFetcher (LocalFetcher.java:copyMapOutput(141)) - localfetcher#27 about to shuffle output of map attempt_local502182601_0034_m_000000_0 decomp: 67 len: 71 to MEMORY
2015-02-12 16:12:53,907 INFO reduce.InMemoryMapOutput (InMemoryMapOutput.java:shuffle(100)) - Read 67 bytes from map-output for attempt_local502182601_0034_m_000000_0
2015-02-12 16:12:53,907 INFO reduce.MergeManagerImpl (MergeManagerImpl.java:closeInMemoryFile(313)) - closeInMemoryFile -> map-output of size: 67, inMemoryMapOutputs.size() -> 1, commitMemory -> 0, usedMemory ->67
2015-02-12 16:12:53,907 INFO reduce.EventFetcher (EventFetcher.java:run(76)) - EventFetcher is interrupted.. Returning
2015-02-12 16:12:53,908 INFO mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) - 1 / 1 copied.
2015-02-12 16:12:53,908 INFO reduce.MergeManagerImpl (MergeManagerImpl.java:finalMerge(673)) - finalMerge called with 1 in-memory map-outputs and 0 on-disk map-outputs
2015-02-12 16:12:53,909 INFO mapred.Merger (Merger.java:merge(606)) - Merging 1 sorted segments
2015-02-12 16:12:53,909 INFO mapred.Merger (Merger.java:merge(705)) - Down to the last merge-pass, with 1 segments left of total size: 58 bytes
2015-02-12 16:12:53,910 INFO reduce.MergeManagerImpl (MergeManagerImpl.java:finalMerge(750)) - Merged 1 segments, 67 bytes to disk to satisfy reduce memory limit
2015-02-12 16:12:53,911 INFO reduce.MergeManagerImpl (MergeManagerImpl.java:finalMerge(780)) - Merging 1 files, 71 bytes from disk
2015-02-12 16:12:53,911 INFO reduce.MergeManagerImpl (MergeManagerImpl.java:finalMerge(795)) - Merging 0 segments, 0 bytes from memory into reduce
2015-02-12 16:12:53,911 INFO mapred.Merger (Merger.java:merge(606)) - Merging 1 sorted segments
2015-02-12 16:12:53,911 INFO mapred.Merger (Merger.java:merge(705)) - Down to the last merge-pass, with 1 segments left of total size: 58 bytes
2015-02-12 16:12:53,912 INFO mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) - 1 / 1 copied.
2015-02-12 16:12:53,912 INFO hadoop.FlowReducer (FlowReducer.java:configure(78)) - cascading version: 2.6.3
2015-02-12 16:12:53,912 INFO hadoop.FlowReducer (FlowReducer.java:configure(79)) - child jvm opts: -Xmx200m
2015-02-12 16:12:53,915 INFO hadoop.FlowReducer (FlowReducer.java:configure(97)) - sourcing from: GroupBy(count-rhs)[by:[{1}:'char']]
2015-02-12 16:12:53,915 INFO hadoop.FlowReducer (FlowReducer.java:configure(100)) - sinking to: TempHfs["SequenceFile[['char', 'count']]"][3761761564/count-rhs/]
2015-02-12 16:12:53,921 INFO mapred.Task (Task.java:done(1001)) - Task:attempt_local502182601_0034_r_000000_0 is done. And is in the process of committing
2015-02-12 16:12:53,922 INFO mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) - 1 / 1 copied.
2015-02-12 16:12:53,923 INFO mapred.Task (Task.java:commit(1162)) - Task attempt_local502182601_0034_r_000000_0 is allowed to commit now
2015-02-12 16:12:53,923 INFO output.FileOutputCommitter (FileOutputCommitter.java:commitTask(439)) - Saved output of task 'attempt_local502182601_0034_r_000000_0' to file:/tmp/hadoop-soberoi/3761761564_count_rhs_F5332AAA9616477EAC6E92C77907B8AC/_temporary/0/task_local502182601_0034_r_000000
2015-02-12 16:12:53,924 INFO mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) - reduce > reduce
2015-02-12 16:12:53,924 INFO mapred.Task (Task.java:sendDone(1121)) - Task 'attempt_local502182601_0034_r_000000_0' done.
2015-02-12 16:12:53,924 INFO mapred.LocalJobRunner (LocalJobRunner.java:run(325)) - Finishing task: attempt_local502182601_0034_r_000000_0
2015-02-12 16:12:53,924 INFO mapred.LocalJobRunner (LocalJobRunner.java:runTasks(456)) - reduce task executor complete.
2015-02-12 16:12:53,933 INFO flow.FlowStep (BaseFlowStep.java:logInfo(834)) - [count-lhs*source*count...] starting step: (2/5)
2015-02-12 16:12:53,933 INFO jvm.JvmMetrics (JvmMetrics.java:init(71)) - Cannot initialize JVM Metrics with processName=JobTracker, sessionId= - already initialized
2015-02-12 16:12:53,934 INFO jvm.JvmMetrics (JvmMetrics.java:init(71)) - Cannot initialize JVM Metrics with processName=JobTracker, sessionId= - already initialized
2015-02-12 16:12:53,966 INFO mapred.FileInputFormat (FileInputFormat.java:listStatus(247)) - Total input paths to process : 1
2015-02-12 16:12:53,977 INFO mapred.FileInputFormat (FileInputFormat.java:listStatus(247)) - Total input paths to process : 1
2015-02-12 16:12:53,993 INFO mapreduce.JobSubmitter (JobSubmitter.java:submitJobInternal(492)) - number of splits:2
2015-02-12 16:12:54,001 INFO Configuration.deprecation (Configuration.java:warnOnceIfDeprecated(1022)) - mapred.used.genericoptionsparser is deprecated. Instead, use mapreduce.client.genericoptionsparser.used
2015-02-12 16:12:54,002 INFO Configuration.deprecation (Configuration.java:warnOnceIfDeprecated(1022)) - mapred.job.tracker is deprecated. Instead, use mapreduce.jobtracker.address
2015-02-12 16:12:54,005 INFO mapreduce.JobSubmitter (JobSubmitter.java:printTokens(575)) - Submitting tokens for job: job_local155775041_0035
2015-02-12 16:12:54,016 WARN conf.Configuration (Configuration.java:loadProperty(2510)) - file:/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/tmp/cascading/staging/soberoi155775041/.staging/job_local155775041_0035/job.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.retry.interval; Ignoring.
2015-02-12 16:12:54,016 WARN conf.Configuration (Configuration.java:loadProperty(2510)) - file:/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/tmp/cascading/staging/soberoi155775041/.staging/job_local155775041_0035/job.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.attempts; Ignoring.
2015-02-12 16:12:54,045 WARN conf.Configuration (Configuration.java:loadProperty(2510)) - file:/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/test/tmp/mapred/local/localRunner/soberoi/job_local155775041_0035/job_local155775041_0035.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.retry.interval; Ignoring.
2015-02-12 16:12:54,046 WARN conf.Configuration (Configuration.java:loadProperty(2510)) - file:/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/test/tmp/mapred/local/localRunner/soberoi/job_local155775041_0035/job_local155775041_0035.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.attempts; Ignoring.
2015-02-12 16:12:54,046 INFO mapreduce.Job (Job.java:submit(1299)) - The url to track the job: http://localhost:8080/
2015-02-12 16:12:54,046 INFO mapred.LocalJobRunner (LocalJobRunner.java:createOutputCommitter(471)) - OutputCommitter set in config null
2015-02-12 16:12:54,047 INFO flow.FlowStep (BaseFlowStep.java:logInfo(834)) - [count-lhs*source*count...] submitted hadoop job: job_local155775041_0035
2015-02-12 16:12:54,047 INFO mapred.LocalJobRunner (LocalJobRunner.java:createOutputCommitter(489)) - OutputCommitter is org.apache.hadoop.mapred.FileOutputCommitter
2015-02-12 16:12:54,047 INFO flow.FlowStep (BaseFlowStep.java:logInfo(834)) - [count-lhs*source*count...] tracking url: http://localhost:8080/
2015-02-12 16:12:54,049 INFO mapred.LocalJobRunner (LocalJobRunner.java:runTasks(448)) - Waiting for map tasks
2015-02-12 16:12:54,049 INFO mapred.LocalJobRunner (LocalJobRunner.java:run(224)) - Starting task: attempt_local155775041_0035_m_000000_0
2015-02-12 16:12:54,050 INFO util.ProcfsBasedProcessTree (ProcfsBasedProcessTree.java:isAvailable(181)) - ProcfsBasedProcessTree currently is supported only on Linux.
2015-02-12 16:12:54,050 INFO mapred.Task (Task.java:initialize(587)) - Using ResourceCalculatorProcessTree : null
2015-02-12 16:12:54,051 INFO io.MultiInputSplit (MultiInputSplit.java:readFields(161)) - current split input path: file:/tmp/hadoop-soberoi/3798598062_source_BF20C2B770DC46CE8090B9EEF941B213/part-00000
2015-02-12 16:12:54,051 INFO mapred.MapTask (MapTask.java:updateJobWithSplit(482)) - Processing split: cascading.tap.hadoop.io.MultiInputSplit@26ef9cf5
2015-02-12 16:12:54,052 INFO mapred.MapTask (MapTask.java:runOldMapper(436)) - numReduceTasks: 1
2015-02-12 16:12:54,067 INFO mapred.MapTask (MapTask.java:setEquator(1202)) - (EQUATOR) 0 kvi 26214396(104857584)
2015-02-12 16:12:54,067 INFO mapred.MapTask (MapTask.java:init(995)) - mapreduce.task.io.sort.mb: 100
2015-02-12 16:12:54,068 INFO mapred.MapTask (MapTask.java:init(996)) - soft limit at 83886080
2015-02-12 16:12:54,068 INFO mapred.MapTask (MapTask.java:init(997)) - bufstart = 0; bufvoid = 104857600
2015-02-12 16:12:54,068 INFO mapred.MapTask (MapTask.java:init(998)) - kvstart = 26214396; length = 6553600
2015-02-12 16:12:54,069 INFO mapred.MapTask (MapTask.java:createSortingCollector(402)) - Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer
2015-02-12 16:12:54,069 INFO hadoop.FlowMapper (FlowMapper.java:configure(71)) - cascading version: 2.6.3
2015-02-12 16:12:54,069 INFO hadoop.FlowMapper (FlowMapper.java:configure(72)) - child jvm opts: -Xmx200m
2015-02-12 16:12:54,072 INFO hadoop.FlowMapper (FlowMapper.java:configure(87)) - sourcing from: TempHfs["SequenceFile[['char', 'count']]"][3798598062/source/]
2015-02-12 16:12:54,073 INFO hadoop.FlowMapper (FlowMapper.java:configure(90)) - sinking to: CoGroup(source*count-rhs)[by: source:[{1}:'char'] count-rhs:[{1}:'char']]
2015-02-12 16:12:54,074 INFO mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) -
2015-02-12 16:12:54,074 INFO mapred.MapTask (MapTask.java:flush(1457)) - Starting flush of map output
2015-02-12 16:12:54,074 INFO mapred.MapTask (MapTask.java:flush(1475)) - Spilling map output
2015-02-12 16:12:54,074 INFO mapred.MapTask (MapTask.java:flush(1476)) - bufstart = 0; bufend = 65; bufvoid = 104857600
2015-02-12 16:12:54,074 INFO mapred.MapTask (MapTask.java:flush(1478)) - kvstart = 26214396(104857584); kvend = 26214380(104857520); length = 17/6553600
2015-02-12 16:12:54,075 INFO mapred.MapTask (MapTask.java:sortAndSpill(1660)) - Finished spill 0
2015-02-12 16:12:54,077 INFO mapred.Task (Task.java:done(1001)) - Task:attempt_local155775041_0035_m_000000_0 is done. And is in the process of committing
2015-02-12 16:12:54,077 INFO mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) - file:/tmp/hadoop-soberoi/3798598062_source_BF20C2B770DC46CE8090B9EEF941B213/part-00000:0+160
2015-02-12 16:12:54,077 INFO mapred.Task (Task.java:sendDone(1121)) - Task 'attempt_local155775041_0035_m_000000_0' done.
2015-02-12 16:12:54,078 INFO mapred.LocalJobRunner (LocalJobRunner.java:run(249)) - Finishing task: attempt_local155775041_0035_m_000000_0
2015-02-12 16:12:54,078 INFO mapred.LocalJobRunner (LocalJobRunner.java:run(224)) - Starting task: attempt_local155775041_0035_m_000001_0
2015-02-12 16:12:54,078 INFO util.ProcfsBasedProcessTree (ProcfsBasedProcessTree.java:isAvailable(181)) - ProcfsBasedProcessTree currently is supported only on Linux.
2015-02-12 16:12:54,079 INFO mapred.Task (Task.java:initialize(587)) - Using ResourceCalculatorProcessTree : null
2015-02-12 16:12:54,079 INFO io.MultiInputSplit (MultiInputSplit.java:readFields(161)) - current split input path: file:/tmp/hadoop-soberoi/3761761564_count_rhs_F5332AAA9616477EAC6E92C77907B8AC/part-00000
2015-02-12 16:12:54,079 INFO mapred.MapTask (MapTask.java:updateJobWithSplit(482)) - Processing split: cascading.tap.hadoop.io.MultiInputSplit@3f078f1c
2015-02-12 16:12:54,080 INFO mapred.MapTask (MapTask.java:runOldMapper(436)) - numReduceTasks: 1
2015-02-12 16:12:54,091 INFO mapred.MapTask (MapTask.java:setEquator(1202)) - (EQUATOR) 0 kvi 26214396(104857584)
2015-02-12 16:12:54,092 INFO mapred.MapTask (MapTask.java:init(995)) - mapreduce.task.io.sort.mb: 100
2015-02-12 16:12:54,092 INFO mapred.MapTask (MapTask.java:init(996)) - soft limit at 83886080
2015-02-12 16:12:54,092 INFO mapred.MapTask (MapTask.java:init(997)) - bufstart = 0; bufvoid = 104857600
2015-02-12 16:12:54,092 INFO mapred.MapTask (MapTask.java:init(998)) - kvstart = 26214396; length = 6553600
2015-02-12 16:12:54,092 INFO mapred.MapTask (MapTask.java:createSortingCollector(402)) - Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer
2015-02-12 16:12:54,093 INFO hadoop.FlowMapper (FlowMapper.java:configure(71)) - cascading version: 2.6.3
2015-02-12 16:12:54,093 INFO hadoop.FlowMapper (FlowMapper.java:configure(72)) - child jvm opts: -Xmx200m
2015-02-12 16:12:54,095 INFO hadoop.FlowMapper (FlowMapper.java:configure(87)) - sourcing from: TempHfs["SequenceFile[['char', 'count']]"][3761761564/count-rhs/]
2015-02-12 16:12:54,095 INFO hadoop.FlowMapper (FlowMapper.java:configure(90)) - sinking to: CoGroup(source*count-rhs)[by: source:[{1}:'char'] count-rhs:[{1}:'char']]
2015-02-12 16:12:54,096 INFO mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) -
2015-02-12 16:12:54,097 INFO mapred.MapTask (MapTask.java:flush(1457)) - Starting flush of map output
2015-02-12 16:12:54,097 INFO mapred.MapTask (MapTask.java:flush(1475)) - Spilling map output
2015-02-12 16:12:54,097 INFO mapred.MapTask (MapTask.java:flush(1476)) - bufstart = 0; bufend = 65; bufvoid = 104857600
2015-02-12 16:12:54,097 INFO mapred.MapTask (MapTask.java:flush(1478)) - kvstart = 26214396(104857584); kvend = 26214380(104857520); length = 17/6553600
2015-02-12 16:12:54,098 INFO mapred.MapTask (MapTask.java:sortAndSpill(1660)) - Finished spill 0
2015-02-12 16:12:54,100 INFO mapred.Task (Task.java:done(1001)) - Task:attempt_local155775041_0035_m_000001_0 is done. And is in the process of committing
2015-02-12 16:12:54,101 INFO mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) - file:/tmp/hadoop-soberoi/3761761564_count_rhs_F5332AAA9616477EAC6E92C77907B8AC/part-00000:0+160
2015-02-12 16:12:54,101 INFO mapred.Task (Task.java:sendDone(1121)) - Task 'attempt_local155775041_0035_m_000001_0' done.
2015-02-12 16:12:54,101 INFO mapred.LocalJobRunner (LocalJobRunner.java:run(249)) - Finishing task: attempt_local155775041_0035_m_000001_0
2015-02-12 16:12:54,101 INFO mapred.LocalJobRunner (LocalJobRunner.java:runTasks(456)) - map task executor complete.
2015-02-12 16:12:54,101 INFO mapred.LocalJobRunner (LocalJobRunner.java:runTasks(448)) - Waiting for reduce tasks
2015-02-12 16:12:54,102 INFO mapred.LocalJobRunner (LocalJobRunner.java:run(302)) - Starting task: attempt_local155775041_0035_r_000000_0
2015-02-12 16:12:54,102 INFO util.ProcfsBasedProcessTree (ProcfsBasedProcessTree.java:isAvailable(181)) - ProcfsBasedProcessTree currently is supported only on Linux.
2015-02-12 16:12:54,103 INFO mapred.Task (Task.java:initialize(587)) - Using ResourceCalculatorProcessTree : null
2015-02-12 16:12:54,103 INFO mapred.ReduceTask (ReduceTask.java:run(362)) - Using ShuffleConsumerPlugin: org.apache.hadoop.mapreduce.task.reduce.Shuffle@38765d
2015-02-12 16:12:54,103 INFO reduce.MergeManagerImpl (MergeManagerImpl.java:<init>(195)) - MergerManager: memoryLimit=493250144, maxSingleShuffleLimit=123312536, mergeThreshold=325545120, ioSortFactor=10, memToMemMergeOutputsThreshold=10
2015-02-12 16:12:54,104 INFO reduce.EventFetcher (EventFetcher.java:run(61)) - attempt_local155775041_0035_r_000000_0 Thread started: EventFetcher for fetching Map Completion Events
2015-02-12 16:12:54,105 INFO reduce.LocalFetcher (LocalFetcher.java:copyMapOutput(141)) - localfetcher#28 about to shuffle output of map attempt_local155775041_0035_m_000000_0 decomp: 77 len: 81 to MEMORY
2015-02-12 16:12:54,105 INFO reduce.InMemoryMapOutput (InMemoryMapOutput.java:shuffle(100)) - Read 77 bytes from map-output for attempt_local155775041_0035_m_000000_0
2015-02-12 16:12:54,105 INFO reduce.MergeManagerImpl (MergeManagerImpl.java:closeInMemoryFile(313)) - closeInMemoryFile -> map-output of size: 77, inMemoryMapOutputs.size() -> 1, commitMemory -> 0, usedMemory ->77
2015-02-12 16:12:54,106 INFO reduce.LocalFetcher (LocalFetcher.java:copyMapOutput(141)) - localfetcher#28 about to shuffle output of map attempt_local155775041_0035_m_000001_0 decomp: 77 len: 81 to MEMORY
2015-02-12 16:12:54,107 INFO reduce.InMemoryMapOutput (InMemoryMapOutput.java:shuffle(100)) - Read 77 bytes from map-output for attempt_local155775041_0035_m_000001_0
2015-02-12 16:12:54,107 INFO reduce.MergeManagerImpl (MergeManagerImpl.java:closeInMemoryFile(313)) - closeInMemoryFile -> map-output of size: 77, inMemoryMapOutputs.size() -> 2, commitMemory -> 77, usedMemory ->154
2015-02-12 16:12:54,107 INFO reduce.EventFetcher (EventFetcher.java:run(76)) - EventFetcher is interrupted.. Returning
2015-02-12 16:12:54,107 INFO mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) - 2 / 2 copied.
2015-02-12 16:12:54,108 INFO reduce.MergeManagerImpl (MergeManagerImpl.java:finalMerge(673)) - finalMerge called with 2 in-memory map-outputs and 0 on-disk map-outputs
2015-02-12 16:12:54,109 INFO mapred.Merger (Merger.java:merge(606)) - Merging 2 sorted segments
2015-02-12 16:12:54,109 INFO mapred.Merger (Merger.java:merge(705)) - Down to the last merge-pass, with 2 segments left of total size: 134 bytes
2015-02-12 16:12:54,111 INFO reduce.MergeManagerImpl (MergeManagerImpl.java:finalMerge(750)) - Merged 2 segments, 154 bytes to disk to satisfy reduce memory limit
2015-02-12 16:12:54,111 INFO reduce.MergeManagerImpl (MergeManagerImpl.java:finalMerge(780)) - Merging 1 files, 156 bytes from disk
2015-02-12 16:12:54,111 INFO reduce.MergeManagerImpl (MergeManagerImpl.java:finalMerge(795)) - Merging 0 segments, 0 bytes from memory into reduce
2015-02-12 16:12:54,111 INFO mapred.Merger (Merger.java:merge(606)) - Merging 1 sorted segments
2015-02-12 16:12:54,111 INFO mapred.Merger (Merger.java:merge(705)) - Down to the last merge-pass, with 1 segments left of total size: 142 bytes
2015-02-12 16:12:54,112 INFO mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) - 2 / 2 copied.
2015-02-12 16:12:54,112 INFO hadoop.FlowReducer (FlowReducer.java:configure(78)) - cascading version: 2.6.3
2015-02-12 16:12:54,112 INFO hadoop.FlowReducer (FlowReducer.java:configure(79)) - child jvm opts: -Xmx200m
2015-02-12 16:12:54,115 INFO hadoop.FlowReducer (FlowReducer.java:configure(97)) - sourcing from: CoGroup(source*count-rhs)[by: source:[{1}:'char'] count-rhs:[{1}:'char']]
2015-02-12 16:12:54,115 INFO hadoop.FlowReducer (FlowReducer.java:configure(100)) - sinking to: TempHfs["SequenceFile[['char', 'num', 'char2', 'count']]"][6501264219/source_count-rhs/]
2015-02-12 16:12:54,125 INFO collect.SpillableTupleList (SpillableTupleList.java:getCodecClass(105)) - attempting to load codec: org.apache.hadoop.io.compress.GzipCodec
2015-02-12 16:12:54,125 INFO collect.SpillableTupleList (SpillableTupleList.java:getCodecClass(110)) - found codec: org.apache.hadoop.io.compress.GzipCodec
2015-02-12 16:12:54,129 INFO mapred.Task (Task.java:done(1001)) - Task:attempt_local155775041_0035_r_000000_0 is done. And is in the process of committing
2015-02-12 16:12:54,130 INFO mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) - 2 / 2 copied.
2015-02-12 16:12:54,130 INFO mapred.Task (Task.java:commit(1162)) - Task attempt_local155775041_0035_r_000000_0 is allowed to commit now
2015-02-12 16:12:54,130 INFO output.FileOutputCommitter (FileOutputCommitter.java:commitTask(439)) - Saved output of task 'attempt_local155775041_0035_r_000000_0' to file:/tmp/hadoop-soberoi/6501264219_source_count_rhs_BC66FC7556A146A5A30829F37762504A/_temporary/0/task_local155775041_0035_r_000000
2015-02-12 16:12:54,131 INFO mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) - reduce > reduce
2015-02-12 16:12:54,131 INFO mapred.Task (Task.java:sendDone(1121)) - Task 'attempt_local155775041_0035_r_000000_0' done.
2015-02-12 16:12:54,131 INFO mapred.LocalJobRunner (LocalJobRunner.java:run(325)) - Finishing task: attempt_local155775041_0035_r_000000_0
2015-02-12 16:12:54,131 INFO mapred.LocalJobRunner (LocalJobRunner.java:runTasks(456)) - reduce task executor complete.
2015-02-12 16:12:54,143 INFO flow.FlowStep (BaseFlowStep.java:logInfo(834)) - [count-lhs*source*count...] starting step: (5/5) ...m/samesourcemergethreeway
2015-02-12 16:12:54,144 INFO jvm.JvmMetrics (JvmMetrics.java:init(71)) - Cannot initialize JVM Metrics with processName=JobTracker, sessionId= - already initialized
2015-02-12 16:12:54,145 INFO jvm.JvmMetrics (JvmMetrics.java:init(71)) - Cannot initialize JVM Metrics with processName=JobTracker, sessionId= - already initialized
2015-02-12 16:12:54,177 INFO mapred.FileInputFormat (FileInputFormat.java:listStatus(247)) - Total input paths to process : 1
2015-02-12 16:12:54,188 INFO mapred.FileInputFormat (FileInputFormat.java:listStatus(247)) - Total input paths to process : 1
2015-02-12 16:12:54,205 INFO mapreduce.JobSubmitter (JobSubmitter.java:submitJobInternal(492)) - number of splits:2
2015-02-12 16:12:54,214 INFO Configuration.deprecation (Configuration.java:warnOnceIfDeprecated(1022)) - mapred.used.genericoptionsparser is deprecated. Instead, use mapreduce.client.genericoptionsparser.used
2015-02-12 16:12:54,214 INFO Configuration.deprecation (Configuration.java:warnOnceIfDeprecated(1022)) - mapred.job.tracker is deprecated. Instead, use mapreduce.jobtracker.address
2015-02-12 16:12:54,217 INFO mapreduce.JobSubmitter (JobSubmitter.java:printTokens(575)) - Submitting tokens for job: job_local1087895190_0036
2015-02-12 16:12:54,227 WARN conf.Configuration (Configuration.java:loadProperty(2510)) - file:/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/tmp/cascading/staging/soberoi1087895190/.staging/job_local1087895190_0036/job.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.retry.interval; Ignoring.
2015-02-12 16:12:54,227 WARN conf.Configuration (Configuration.java:loadProperty(2510)) - file:/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/tmp/cascading/staging/soberoi1087895190/.staging/job_local1087895190_0036/job.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.attempts; Ignoring.
2015-02-12 16:12:54,261 WARN conf.Configuration (Configuration.java:loadProperty(2510)) - file:/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/test/tmp/mapred/local/localRunner/soberoi/job_local1087895190_0036/job_local1087895190_0036.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.retry.interval; Ignoring.
2015-02-12 16:12:54,262 WARN conf.Configuration (Configuration.java:loadProperty(2510)) - file:/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/test/tmp/mapred/local/localRunner/soberoi/job_local1087895190_0036/job_local1087895190_0036.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.attempts; Ignoring.
2015-02-12 16:12:54,262 INFO mapreduce.Job (Job.java:submit(1299)) - The url to track the job: http://localhost:8080/
2015-02-12 16:12:54,263 INFO mapred.LocalJobRunner (LocalJobRunner.java:createOutputCommitter(471)) - OutputCommitter set in config null
2015-02-12 16:12:54,263 INFO flow.FlowStep (BaseFlowStep.java:logInfo(834)) - [count-lhs*source*count...] submitted hadoop job: job_local1087895190_0036
2015-02-12 16:12:54,263 INFO mapred.LocalJobRunner (LocalJobRunner.java:createOutputCommitter(489)) - OutputCommitter is org.apache.hadoop.mapred.FileOutputCommitter
2015-02-12 16:12:54,263 INFO flow.FlowStep (BaseFlowStep.java:logInfo(834)) - [count-lhs*source*count...] tracking url: http://localhost:8080/
2015-02-12 16:12:54,265 INFO mapred.LocalJobRunner (LocalJobRunner.java:runTasks(448)) - Waiting for map tasks
2015-02-12 16:12:54,265 INFO mapred.LocalJobRunner (LocalJobRunner.java:run(224)) - Starting task: attempt_local1087895190_0036_m_000000_0
2015-02-12 16:12:54,266 INFO util.ProcfsBasedProcessTree (ProcfsBasedProcessTree.java:isAvailable(181)) - ProcfsBasedProcessTree currently is supported only on Linux.
2015-02-12 16:12:54,266 INFO mapred.Task (Task.java:initialize(587)) - Using ResourceCalculatorProcessTree : null
2015-02-12 16:12:54,266 INFO io.MultiInputSplit (MultiInputSplit.java:readFields(161)) - current split input path: file:/tmp/hadoop-soberoi/6501264219_source_count_rhs_BC66FC7556A146A5A30829F37762504A/part-00000
2015-02-12 16:12:54,267 INFO mapred.MapTask (MapTask.java:updateJobWithSplit(482)) - Processing split: cascading.tap.hadoop.io.MultiInputSplit@39d47789
2015-02-12 16:12:54,267 INFO mapred.MapTask (MapTask.java:runOldMapper(436)) - numReduceTasks: 1
2015-02-12 16:12:54,278 INFO mapred.MapTask (MapTask.java:setEquator(1202)) - (EQUATOR) 0 kvi 26214396(104857584)
2015-02-12 16:12:54,278 INFO mapred.MapTask (MapTask.java:init(995)) - mapreduce.task.io.sort.mb: 100
2015-02-12 16:12:54,278 INFO mapred.MapTask (MapTask.java:init(996)) - soft limit at 83886080
2015-02-12 16:12:54,278 INFO mapred.MapTask (MapTask.java:init(997)) - bufstart = 0; bufvoid = 104857600
2015-02-12 16:12:54,278 INFO mapred.MapTask (MapTask.java:init(998)) - kvstart = 26214396; length = 6553600
2015-02-12 16:12:54,279 INFO mapred.MapTask (MapTask.java:createSortingCollector(402)) - Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer
2015-02-12 16:12:54,279 INFO hadoop.FlowMapper (FlowMapper.java:configure(71)) - cascading version: 2.6.3
2015-02-12 16:12:54,279 INFO hadoop.FlowMapper (FlowMapper.java:configure(72)) - child jvm opts: -Xmx200m
2015-02-12 16:12:54,283 INFO hadoop.FlowMapper (FlowMapper.java:configure(87)) - sourcing from: TempHfs["SequenceFile[['char', 'num', 'char2', 'count']]"][6501264219/source_count-rhs/]
2015-02-12 16:12:54,283 INFO hadoop.FlowMapper (FlowMapper.java:configure(90)) - sinking to: CoGroup(count-lhs*source*count-rhs)[by: count-lhs:[{1}:'char'] source*count-rhs:[{1}:'char']]
2015-02-12 16:12:54,284 INFO mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) -
2015-02-12 16:12:54,284 INFO mapred.MapTask (MapTask.java:flush(1457)) - Starting flush of map output
2015-02-12 16:12:54,284 INFO mapred.MapTask (MapTask.java:flush(1475)) - Spilling map output
2015-02-12 16:12:54,284 INFO mapred.MapTask (MapTask.java:flush(1476)) - bufstart = 0; bufend = 105; bufvoid = 104857600
2015-02-12 16:12:54,284 INFO mapred.MapTask (MapTask.java:flush(1478)) - kvstart = 26214396(104857584); kvend = 26214380(104857520); length = 17/6553600
2015-02-12 16:12:54,286 INFO mapred.MapTask (MapTask.java:sortAndSpill(1660)) - Finished spill 0
2015-02-12 16:12:54,287 INFO mapred.Task (Task.java:done(1001)) - Task:attempt_local1087895190_0036_m_000000_0 is done. And is in the process of committing
2015-02-12 16:12:54,288 INFO mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) - file:/tmp/hadoop-soberoi/6501264219_source_count_rhs_BC66FC7556A146A5A30829F37762504A/part-00000:0+200
2015-02-12 16:12:54,288 INFO mapred.Task (Task.java:sendDone(1121)) - Task 'attempt_local1087895190_0036_m_000000_0' done.
2015-02-12 16:12:54,288 INFO mapred.LocalJobRunner (LocalJobRunner.java:run(249)) - Finishing task: attempt_local1087895190_0036_m_000000_0
2015-02-12 16:12:54,288 INFO mapred.LocalJobRunner (LocalJobRunner.java:run(224)) - Starting task: attempt_local1087895190_0036_m_000001_0
2015-02-12 16:12:54,289 INFO util.ProcfsBasedProcessTree (ProcfsBasedProcessTree.java:isAvailable(181)) - ProcfsBasedProcessTree currently is supported only on Linux.
2015-02-12 16:12:54,289 INFO mapred.Task (Task.java:initialize(587)) - Using ResourceCalculatorProcessTree : null
2015-02-12 16:12:54,290 INFO io.MultiInputSplit (MultiInputSplit.java:readFields(161)) - current split input path: file:/tmp/hadoop-soberoi/6668818197_count_lhs_E0299A91C9AD4B61A1E657A2086B2F4E/part-00000
2015-02-12 16:12:54,290 INFO mapred.MapTask (MapTask.java:updateJobWithSplit(482)) - Processing split: cascading.tap.hadoop.io.MultiInputSplit@639f8c4a
2015-02-12 16:12:54,291 INFO mapred.MapTask (MapTask.java:runOldMapper(436)) - numReduceTasks: 1
2015-02-12 16:12:54,302 INFO mapred.MapTask (MapTask.java:setEquator(1202)) - (EQUATOR) 0 kvi 26214396(104857584)
2015-02-12 16:12:54,302 INFO mapred.MapTask (MapTask.java:init(995)) - mapreduce.task.io.sort.mb: 100
2015-02-12 16:12:54,302 INFO mapred.MapTask (MapTask.java:init(996)) - soft limit at 83886080
2015-02-12 16:12:54,302 INFO mapred.MapTask (MapTask.java:init(997)) - bufstart = 0; bufvoid = 104857600
2015-02-12 16:12:54,302 INFO mapred.MapTask (MapTask.java:init(998)) - kvstart = 26214396; length = 6553600
2015-02-12 16:12:54,303 INFO mapred.MapTask (MapTask.java:createSortingCollector(402)) - Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer
2015-02-12 16:12:54,303 INFO hadoop.FlowMapper (FlowMapper.java:configure(71)) - cascading version: 2.6.3
2015-02-12 16:12:54,303 INFO hadoop.FlowMapper (FlowMapper.java:configure(72)) - child jvm opts: -Xmx200m
2015-02-12 16:12:54,306 INFO hadoop.FlowMapper (FlowMapper.java:configure(87)) - sourcing from: TempHfs["SequenceFile[['char', 'count']]"][6668818197/count-lhs/]
2015-02-12 16:12:54,306 INFO hadoop.FlowMapper (FlowMapper.java:configure(90)) - sinking to: CoGroup(count-lhs*source*count-rhs)[by: count-lhs:[{1}:'char'] source*count-rhs:[{1}:'char']]
2015-02-12 16:12:54,308 INFO mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) -
2015-02-12 16:12:54,308 INFO mapred.MapTask (MapTask.java:flush(1457)) - Starting flush of map output
2015-02-12 16:12:54,308 INFO mapred.MapTask (MapTask.java:flush(1475)) - Spilling map output
2015-02-12 16:12:54,308 INFO mapred.MapTask (MapTask.java:flush(1476)) - bufstart = 0; bufend = 65; bufvoid = 104857600
2015-02-12 16:12:54,308 INFO mapred.MapTask (MapTask.java:flush(1478)) - kvstart = 26214396(104857584); kvend = 26214380(104857520); length = 17/6553600
2015-02-12 16:12:54,309 INFO mapred.MapTask (MapTask.java:sortAndSpill(1660)) - Finished spill 0
2015-02-12 16:12:54,310 INFO mapred.Task (Task.java:done(1001)) - Task:attempt_local1087895190_0036_m_000001_0 is done. And is in the process of committing
2015-02-12 16:12:54,311 INFO mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) - file:/tmp/hadoop-soberoi/6668818197_count_lhs_E0299A91C9AD4B61A1E657A2086B2F4E/part-00000:0+160
2015-02-12 16:12:54,311 INFO mapred.Task (Task.java:sendDone(1121)) - Task 'attempt_local1087895190_0036_m_000001_0' done.
2015-02-12 16:12:54,311 INFO mapred.LocalJobRunner (LocalJobRunner.java:run(249)) - Finishing task: attempt_local1087895190_0036_m_000001_0
2015-02-12 16:12:54,311 INFO mapred.LocalJobRunner (LocalJobRunner.java:runTasks(456)) - map task executor complete.
2015-02-12 16:12:54,312 INFO mapred.LocalJobRunner (LocalJobRunner.java:runTasks(448)) - Waiting for reduce tasks
2015-02-12 16:12:54,312 INFO mapred.LocalJobRunner (LocalJobRunner.java:run(302)) - Starting task: attempt_local1087895190_0036_r_000000_0
2015-02-12 16:12:54,313 INFO util.ProcfsBasedProcessTree (ProcfsBasedProcessTree.java:isAvailable(181)) - ProcfsBasedProcessTree currently is supported only on Linux.
2015-02-12 16:12:54,313 INFO mapred.Task (Task.java:initialize(587)) - Using ResourceCalculatorProcessTree : null
2015-02-12 16:12:54,313 INFO mapred.ReduceTask (ReduceTask.java:run(362)) - Using ShuffleConsumerPlugin: org.apache.hadoop.mapreduce.task.reduce.Shuffle@570fd91e
2015-02-12 16:12:54,314 INFO reduce.MergeManagerImpl (MergeManagerImpl.java:<init>(195)) - MergerManager: memoryLimit=493250144, maxSingleShuffleLimit=123312536, mergeThreshold=325545120, ioSortFactor=10, memToMemMergeOutputsThreshold=10
2015-02-12 16:12:54,315 INFO reduce.EventFetcher (EventFetcher.java:run(61)) - attempt_local1087895190_0036_r_000000_0 Thread started: EventFetcher for fetching Map Completion Events
2015-02-12 16:12:54,316 INFO reduce.LocalFetcher (LocalFetcher.java:copyMapOutput(141)) - localfetcher#29 about to shuffle output of map attempt_local1087895190_0036_m_000001_0 decomp: 77 len: 81 to MEMORY
2015-02-12 16:12:54,316 INFO reduce.InMemoryMapOutput (InMemoryMapOutput.java:shuffle(100)) - Read 77 bytes from map-output for attempt_local1087895190_0036_m_000001_0
2015-02-12 16:12:54,316 INFO reduce.MergeManagerImpl (MergeManagerImpl.java:closeInMemoryFile(313)) - closeInMemoryFile -> map-output of size: 77, inMemoryMapOutputs.size() -> 1, commitMemory -> 0, usedMemory ->77
2015-02-12 16:12:54,317 INFO reduce.LocalFetcher (LocalFetcher.java:copyMapOutput(141)) - localfetcher#29 about to shuffle output of map attempt_local1087895190_0036_m_000000_0 decomp: 117 len: 121 to MEMORY
2015-02-12 16:12:54,317 INFO reduce.InMemoryMapOutput (InMemoryMapOutput.java:shuffle(100)) - Read 117 bytes from map-output for attempt_local1087895190_0036_m_000000_0
2015-02-12 16:12:54,317 INFO reduce.MergeManagerImpl (MergeManagerImpl.java:closeInMemoryFile(313)) - closeInMemoryFile -> map-output of size: 117, inMemoryMapOutputs.size() -> 2, commitMemory -> 77, usedMemory ->194
2015-02-12 16:12:54,317 INFO reduce.EventFetcher (EventFetcher.java:run(76)) - EventFetcher is interrupted.. Returning
2015-02-12 16:12:54,318 INFO mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) - 2 / 2 copied.
2015-02-12 16:12:54,318 INFO reduce.MergeManagerImpl (MergeManagerImpl.java:finalMerge(673)) - finalMerge called with 2 in-memory map-outputs and 0 on-disk map-outputs
2015-02-12 16:12:54,319 INFO mapred.Merger (Merger.java:merge(606)) - Merging 2 sorted segments
2015-02-12 16:12:54,319 INFO mapred.Merger (Merger.java:merge(705)) - Down to the last merge-pass, with 2 segments left of total size: 174 bytes
2015-02-12 16:12:54,320 INFO reduce.MergeManagerImpl (MergeManagerImpl.java:finalMerge(750)) - Merged 2 segments, 194 bytes to disk to satisfy reduce memory limit
2015-02-12 16:12:54,321 INFO reduce.MergeManagerImpl (MergeManagerImpl.java:finalMerge(780)) - Merging 1 files, 196 bytes from disk
2015-02-12 16:12:54,321 INFO reduce.MergeManagerImpl (MergeManagerImpl.java:finalMerge(795)) - Merging 0 segments, 0 bytes from memory into reduce
2015-02-12 16:12:54,321 INFO mapred.Merger (Merger.java:merge(606)) - Merging 1 sorted segments
2015-02-12 16:12:54,321 INFO mapred.Merger (Merger.java:merge(705)) - Down to the last merge-pass, with 1 segments left of total size: 182 bytes
2015-02-12 16:12:54,321 INFO mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) - 2 / 2 copied.
2015-02-12 16:12:54,322 INFO hadoop.FlowReducer (FlowReducer.java:configure(78)) - cascading version: 2.6.3
2015-02-12 16:12:54,322 INFO hadoop.FlowReducer (FlowReducer.java:configure(79)) - child jvm opts: -Xmx200m
2015-02-12 16:12:54,325 INFO hadoop.FlowReducer (FlowReducer.java:configure(97)) - sourcing from: CoGroup(count-lhs*source*count-rhs)[by: count-lhs:[{1}:'char'] source*count-rhs:[{1}:'char']]
2015-02-12 16:12:54,325 INFO hadoop.FlowReducer (FlowReducer.java:configure(100)) - sinking to: Hfs["TextDelimited[['char', 'count']]"]["/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/test/output/hadoop2-mr1/assemblyhelpersplatform/samesourcemergethreeway"]
2015-02-12 16:12:54,331 INFO collect.SpillableTupleList (SpillableTupleList.java:getCodecClass(105)) - attempting to load codec: org.apache.hadoop.io.compress.GzipCodec
2015-02-12 16:12:54,331 INFO collect.SpillableTupleList (SpillableTupleList.java:getCodecClass(110)) - found codec: org.apache.hadoop.io.compress.GzipCodec
2015-02-12 16:12:54,333 INFO mapred.Task (Task.java:done(1001)) - Task:attempt_local1087895190_0036_r_000000_0 is done. And is in the process of committing
2015-02-12 16:12:54,334 INFO mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) - 2 / 2 copied.
2015-02-12 16:12:54,334 INFO mapred.Task (Task.java:commit(1162)) - Task attempt_local1087895190_0036_r_000000_0 is allowed to commit now
2015-02-12 16:12:54,335 INFO output.FileOutputCommitter (FileOutputCommitter.java:commitTask(439)) - Saved output of task 'attempt_local1087895190_0036_r_000000_0' to file:/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/test/output/hadoop2-mr1/assemblyhelpersplatform/samesourcemergethreeway/_temporary/0/task_local1087895190_0036_r_000000
2015-02-12 16:12:54,336 INFO mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) - reduce > reduce
2015-02-12 16:12:54,336 INFO mapred.Task (Task.java:sendDone(1121)) - Task 'attempt_local1087895190_0036_r_000000_0' done.
2015-02-12 16:12:54,336 INFO mapred.LocalJobRunner (LocalJobRunner.java:run(325)) - Finishing task: attempt_local1087895190_0036_r_000000_0
2015-02-12 16:12:54,336 INFO mapred.LocalJobRunner (LocalJobRunner.java:runTasks(456)) - reduce task executor complete.
2015-02-12 16:12:54,359 INFO util.Hadoop18TapUtil (Hadoop18TapUtil.java:cleanTempPath(219)) - deleting temp path /Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/test/output/hadoop2-mr1/assemblyhelpersplatform/samesourcemergethreeway/_temporary
2015-02-12 16:12:54,378 INFO mapred.FileInputFormat (FileInputFormat.java:listStatus(247)) - Total input paths to process : 1
2015-02-12 16:12:54,402 INFO mapred.FileInputFormat (FileInputFormat.java:listStatus(247)) - Total input paths to process : 1
2015-02-12 16:12:54,403 INFO cascading.PlatformTestCase (PlatformTestCase.java:tearDown(209)) - copying to local /Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/test/output//hadoop2-mr1/assemblyhelpersplatform/samesourcemergethreeway
2015-02-12 16:12:54,415 INFO util.HadoopUtil (HadoopUtil.java:findMainClass(336)) - using default application jar, may cause class not found exceptions on the cluster
2015-02-12 16:12:54,415 INFO planner.HadoopPlanner (HadoopPlanner.java:initialize(225)) - using application jar: /Users/soberoi/.gradle/caches/modules-2/files-2.1/cascading/cascading-hadoop2-mr1/2.6.3/9ae016e2b8d246b9c1de9168e715d9f1b971aa4a/cascading-hadoop2-mr1-2.6.3.jar
2015-02-12 16:12:54,419 INFO Configuration.deprecation (Configuration.java:warnOnceIfDeprecated(1022)) - mapred.used.genericoptionsparser is deprecated. Instead, use mapreduce.client.genericoptionsparser.used
2015-02-12 16:12:54,420 INFO Configuration.deprecation (Configuration.java:warnOnceIfDeprecated(1022)) - mapred.job.tracker is deprecated. Instead, use mapreduce.jobtracker.address
2015-02-12 16:12:54,420 INFO Configuration.deprecation (Configuration.java:warnOnceIfDeprecated(1022)) - mapred.output.compress is deprecated. Instead, use mapreduce.output.fileoutputformat.compress
2015-02-12 16:12:54,425 INFO flow.Flow (BaseFlow.java:logInfo(1378)) - [lhs+rhs] starting
2015-02-12 16:12:54,426 INFO flow.Flow (BaseFlow.java:logInfo(1378)) - [lhs+rhs] source: Hfs["TextLine[['offset', 'line']->[ALL]]"]["/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/classes/test/data/rhs.txt"]
2015-02-12 16:12:54,426 INFO flow.Flow (BaseFlow.java:logInfo(1378)) - [lhs+rhs] source: Hfs["TextLine[['offset', 'line']->[ALL]]"]["/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/classes/test/data/lhs.txt"]
2015-02-12 16:12:54,426 INFO flow.Flow (BaseFlow.java:logInfo(1378)) - [lhs+rhs] sink: Hfs["TextLine[['item']->['num', 'char']]"]["/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/test/output/hadoop2-mr1/assemblyhelpersplatform/uniquemerge-nondeterministic"]
2015-02-12 16:12:54,426 INFO flow.Flow (BaseFlow.java:logInfo(1378)) - [lhs+rhs] parallel execution is enabled: false
2015-02-12 16:12:54,426 INFO flow.Flow (BaseFlow.java:logInfo(1378)) - [lhs+rhs] starting jobs: 1
2015-02-12 16:12:54,426 INFO flow.Flow (BaseFlow.java:logInfo(1378)) - [lhs+rhs] allocating threads: 1
2015-02-12 16:12:54,426 INFO flow.FlowStep (BaseFlowStep.java:logInfo(834)) - [lhs+rhs] starting step: (1/1) ...quemerge-nondeterministic
2015-02-12 16:12:54,427 INFO jvm.JvmMetrics (JvmMetrics.java:init(71)) - Cannot initialize JVM Metrics with processName=JobTracker, sessionId= - already initialized
2015-02-12 16:12:54,427 INFO jvm.JvmMetrics (JvmMetrics.java:init(71)) - Cannot initialize JVM Metrics with processName=JobTracker, sessionId= - already initialized
2015-02-12 16:12:54,444 INFO mapred.FileInputFormat (FileInputFormat.java:listStatus(247)) - Total input paths to process : 1
2015-02-12 16:12:54,445 INFO mapred.FileInputFormat (FileInputFormat.java:listStatus(247)) - Total input paths to process : 1
2015-02-12 16:12:54,461 INFO mapreduce.JobSubmitter (JobSubmitter.java:submitJobInternal(492)) - number of splits:2
2015-02-12 16:12:54,472 INFO mapreduce.JobSubmitter (JobSubmitter.java:printTokens(575)) - Submitting tokens for job: job_local1683387041_0037
2015-02-12 16:12:54,481 WARN conf.Configuration (Configuration.java:loadProperty(2510)) - file:/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/tmp/cascading/staging/soberoi1683387041/.staging/job_local1683387041_0037/job.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.retry.interval; Ignoring.
2015-02-12 16:12:54,482 WARN conf.Configuration (Configuration.java:loadProperty(2510)) - file:/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/tmp/cascading/staging/soberoi1683387041/.staging/job_local1683387041_0037/job.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.attempts; Ignoring.
2015-02-12 16:12:54,508 WARN conf.Configuration (Configuration.java:loadProperty(2510)) - file:/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/test/tmp/mapred/local/localRunner/soberoi/job_local1683387041_0037/job_local1683387041_0037.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.retry.interval; Ignoring.
2015-02-12 16:12:54,509 WARN conf.Configuration (Configuration.java:loadProperty(2510)) - file:/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/test/tmp/mapred/local/localRunner/soberoi/job_local1683387041_0037/job_local1683387041_0037.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.attempts; Ignoring.
2015-02-12 16:12:54,510 INFO mapreduce.Job (Job.java:submit(1299)) - The url to track the job: http://localhost:8080/
2015-02-12 16:12:54,510 INFO flow.FlowStep (BaseFlowStep.java:logInfo(834)) - [lhs+rhs] submitted hadoop job: job_local1683387041_0037
2015-02-12 16:12:54,510 INFO mapred.LocalJobRunner (LocalJobRunner.java:createOutputCommitter(471)) - OutputCommitter set in config null
2015-02-12 16:12:54,510 INFO mapred.LocalJobRunner (LocalJobRunner.java:createOutputCommitter(489)) - OutputCommitter is org.apache.hadoop.mapred.FileOutputCommitter
2015-02-12 16:12:54,510 INFO flow.FlowStep (BaseFlowStep.java:logInfo(834)) - [lhs+rhs] tracking url: http://localhost:8080/
2015-02-12 16:12:54,512 INFO mapred.LocalJobRunner (LocalJobRunner.java:runTasks(448)) - Waiting for map tasks
2015-02-12 16:12:54,512 INFO mapred.LocalJobRunner (LocalJobRunner.java:run(224)) - Starting task: attempt_local1683387041_0037_m_000000_0
2015-02-12 16:12:54,513 INFO util.ProcfsBasedProcessTree (ProcfsBasedProcessTree.java:isAvailable(181)) - ProcfsBasedProcessTree currently is supported only on Linux.
2015-02-12 16:12:54,513 INFO mapred.Task (Task.java:initialize(587)) - Using ResourceCalculatorProcessTree : null
2015-02-12 16:12:54,513 INFO io.MultiInputSplit (MultiInputSplit.java:readFields(161)) - current split input path: file:/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/classes/test/data/lhs.txt
2015-02-12 16:12:54,514 INFO mapred.MapTask (MapTask.java:updateJobWithSplit(482)) - Processing split: cascading.tap.hadoop.io.MultiInputSplit@65b2b2f0
2015-02-12 16:12:54,514 INFO mapred.MapTask (MapTask.java:runOldMapper(436)) - numReduceTasks: 1
2015-02-12 16:12:54,525 INFO mapred.MapTask (MapTask.java:setEquator(1202)) - (EQUATOR) 0 kvi 26214396(104857584)
2015-02-12 16:12:54,525 INFO mapred.MapTask (MapTask.java:init(995)) - mapreduce.task.io.sort.mb: 100
2015-02-12 16:12:54,525 INFO mapred.MapTask (MapTask.java:init(996)) - soft limit at 83886080
2015-02-12 16:12:54,525 INFO mapred.MapTask (MapTask.java:init(997)) - bufstart = 0; bufvoid = 104857600
2015-02-12 16:12:54,525 INFO mapred.MapTask (MapTask.java:init(998)) - kvstart = 26214396; length = 6553600
2015-02-12 16:12:54,526 INFO mapred.MapTask (MapTask.java:createSortingCollector(402)) - Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer
2015-02-12 16:12:54,526 INFO hadoop.FlowMapper (FlowMapper.java:configure(71)) - cascading version: 2.6.3
2015-02-12 16:12:54,526 INFO hadoop.FlowMapper (FlowMapper.java:configure(72)) - child jvm opts: -Xmx200m
2015-02-12 16:12:54,528 INFO hadoop.FlowMapper (FlowMapper.java:configure(87)) - sourcing from: Hfs["TextLine[['offset', 'line']->[ALL]]"]["/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/classes/test/data/lhs.txt"]
2015-02-12 16:12:54,529 INFO hadoop.FlowMapper (FlowMapper.java:configure(90)) - sinking to: GroupBy(lhs+rhs)[by: lhs:[{1}:'num'] rhs:[{1}:'num']]
2015-02-12 16:12:54,532 INFO mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) -
2015-02-12 16:12:54,532 INFO mapred.MapTask (MapTask.java:flush(1457)) - Starting flush of map output
2015-02-12 16:12:54,532 INFO mapred.MapTask (MapTask.java:flush(1475)) - Spilling map output
2015-02-12 16:12:54,532 INFO mapred.MapTask (MapTask.java:flush(1476)) - bufstart = 0; bufend = 75; bufvoid = 104857600
2015-02-12 16:12:54,532 INFO mapred.MapTask (MapTask.java:flush(1478)) - kvstart = 26214396(104857584); kvend = 26214380(104857520); length = 17/6553600
2015-02-12 16:12:54,534 INFO mapred.MapTask (MapTask.java:sortAndSpill(1660)) - Finished spill 0
2015-02-12 16:12:54,535 INFO mapred.Task (Task.java:done(1001)) - Task:attempt_local1683387041_0037_m_000000_0 is done. And is in the process of committing
2015-02-12 16:12:54,536 INFO mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) - file:/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/classes/test/data/lhs.txt:0+52
2015-02-12 16:12:54,536 INFO mapred.Task (Task.java:sendDone(1121)) - Task 'attempt_local1683387041_0037_m_000000_0' done.
2015-02-12 16:12:54,536 INFO mapred.LocalJobRunner (LocalJobRunner.java:run(249)) - Finishing task: attempt_local1683387041_0037_m_000000_0
2015-02-12 16:12:54,536 INFO mapred.LocalJobRunner (LocalJobRunner.java:run(224)) - Starting task: attempt_local1683387041_0037_m_000001_0
2015-02-12 16:12:54,537 INFO util.ProcfsBasedProcessTree (ProcfsBasedProcessTree.java:isAvailable(181)) - ProcfsBasedProcessTree currently is supported only on Linux.
2015-02-12 16:12:54,537 INFO mapred.Task (Task.java:initialize(587)) - Using ResourceCalculatorProcessTree : null
2015-02-12 16:12:54,538 INFO io.MultiInputSplit (MultiInputSplit.java:readFields(161)) - current split input path: file:/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/classes/test/data/rhs.txt
2015-02-12 16:12:54,538 INFO mapred.MapTask (MapTask.java:updateJobWithSplit(482)) - Processing split: cascading.tap.hadoop.io.MultiInputSplit@574e6595
2015-02-12 16:12:54,539 INFO mapred.MapTask (MapTask.java:runOldMapper(436)) - numReduceTasks: 1
2015-02-12 16:12:54,555 INFO mapred.MapTask (MapTask.java:setEquator(1202)) - (EQUATOR) 0 kvi 26214396(104857584)
2015-02-12 16:12:54,555 INFO mapred.MapTask (MapTask.java:init(995)) - mapreduce.task.io.sort.mb: 100
2015-02-12 16:12:54,555 INFO mapred.MapTask (MapTask.java:init(996)) - soft limit at 83886080
2015-02-12 16:12:54,555 INFO mapred.MapTask (MapTask.java:init(997)) - bufstart = 0; bufvoid = 104857600
2015-02-12 16:12:54,555 INFO mapred.MapTask (MapTask.java:init(998)) - kvstart = 26214396; length = 6553600
2015-02-12 16:12:54,556 INFO mapred.MapTask (MapTask.java:createSortingCollector(402)) - Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer
2015-02-12 16:12:54,556 INFO hadoop.FlowMapper (FlowMapper.java:configure(71)) - cascading version: 2.6.3
2015-02-12 16:12:54,556 INFO hadoop.FlowMapper (FlowMapper.java:configure(72)) - child jvm opts: -Xmx200m
2015-02-12 16:12:54,559 INFO hadoop.FlowMapper (FlowMapper.java:configure(87)) - sourcing from: Hfs["TextLine[['offset', 'line']->[ALL]]"]["/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/classes/test/data/rhs.txt"]
2015-02-12 16:12:54,559 INFO hadoop.FlowMapper (FlowMapper.java:configure(90)) - sinking to: GroupBy(lhs+rhs)[by: lhs:[{1}:'num'] rhs:[{1}:'num']]
2015-02-12 16:12:54,561 INFO mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) -
2015-02-12 16:12:54,561 INFO mapred.MapTask (MapTask.java:flush(1457)) - Starting flush of map output
2015-02-12 16:12:54,561 INFO mapred.MapTask (MapTask.java:flush(1475)) - Spilling map output
2015-02-12 16:12:54,561 INFO mapred.MapTask (MapTask.java:flush(1476)) - bufstart = 0; bufend = 75; bufvoid = 104857600
2015-02-12 16:12:54,561 INFO mapred.MapTask (MapTask.java:flush(1478)) - kvstart = 26214396(104857584); kvend = 26214380(104857520); length = 17/6553600
2015-02-12 16:12:54,563 INFO mapred.MapTask (MapTask.java:sortAndSpill(1660)) - Finished spill 0
2015-02-12 16:12:54,565 INFO mapred.Task (Task.java:done(1001)) - Task:attempt_local1683387041_0037_m_000001_0 is done. And is in the process of committing
2015-02-12 16:12:54,566 INFO mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) - file:/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/classes/test/data/rhs.txt:0+51
2015-02-12 16:12:54,566 INFO mapred.Task (Task.java:sendDone(1121)) - Task 'attempt_local1683387041_0037_m_000001_0' done.
2015-02-12 16:12:54,566 INFO mapred.LocalJobRunner (LocalJobRunner.java:run(249)) - Finishing task: attempt_local1683387041_0037_m_000001_0
2015-02-12 16:12:54,567 INFO mapred.LocalJobRunner (LocalJobRunner.java:runTasks(456)) - map task executor complete.
2015-02-12 16:12:54,567 INFO mapred.LocalJobRunner (LocalJobRunner.java:runTasks(448)) - Waiting for reduce tasks
2015-02-12 16:12:54,567 INFO mapred.LocalJobRunner (LocalJobRunner.java:run(302)) - Starting task: attempt_local1683387041_0037_r_000000_0
2015-02-12 16:12:54,568 INFO util.ProcfsBasedProcessTree (ProcfsBasedProcessTree.java:isAvailable(181)) - ProcfsBasedProcessTree currently is supported only on Linux.
2015-02-12 16:12:54,569 INFO mapred.Task (Task.java:initialize(587)) - Using ResourceCalculatorProcessTree : null
2015-02-12 16:12:54,569 INFO mapred.ReduceTask (ReduceTask.java:run(362)) - Using ShuffleConsumerPlugin: org.apache.hadoop.mapreduce.task.reduce.Shuffle@5fe382df
2015-02-12 16:12:54,569 INFO reduce.MergeManagerImpl (MergeManagerImpl.java:<init>(195)) - MergerManager: memoryLimit=493250144, maxSingleShuffleLimit=123312536, mergeThreshold=325545120, ioSortFactor=10, memToMemMergeOutputsThreshold=10
2015-02-12 16:12:54,570 INFO reduce.EventFetcher (EventFetcher.java:run(61)) - attempt_local1683387041_0037_r_000000_0 Thread started: EventFetcher for fetching Map Completion Events
2015-02-12 16:12:54,570 INFO reduce.LocalFetcher (LocalFetcher.java:copyMapOutput(141)) - localfetcher#30 about to shuffle output of map attempt_local1683387041_0037_m_000001_0 decomp: 87 len: 91 to MEMORY
2015-02-12 16:12:54,571 INFO reduce.InMemoryMapOutput (InMemoryMapOutput.java:shuffle(100)) - Read 87 bytes from map-output for attempt_local1683387041_0037_m_000001_0
2015-02-12 16:12:54,571 INFO reduce.MergeManagerImpl (MergeManagerImpl.java:closeInMemoryFile(313)) - closeInMemoryFile -> map-output of size: 87, inMemoryMapOutputs.size() -> 1, commitMemory -> 0, usedMemory ->87
2015-02-12 16:12:54,572 INFO reduce.LocalFetcher (LocalFetcher.java:copyMapOutput(141)) - localfetcher#30 about to shuffle output of map attempt_local1683387041_0037_m_000000_0 decomp: 87 len: 91 to MEMORY
2015-02-12 16:12:54,573 INFO reduce.InMemoryMapOutput (InMemoryMapOutput.java:shuffle(100)) - Read 87 bytes from map-output for attempt_local1683387041_0037_m_000000_0
2015-02-12 16:12:54,573 INFO reduce.MergeManagerImpl (MergeManagerImpl.java:closeInMemoryFile(313)) - closeInMemoryFile -> map-output of size: 87, inMemoryMapOutputs.size() -> 2, commitMemory -> 87, usedMemory ->174
2015-02-12 16:12:54,573 INFO reduce.EventFetcher (EventFetcher.java:run(76)) - EventFetcher is interrupted.. Returning
2015-02-12 16:12:54,574 INFO mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) - 2 / 2 copied.
2015-02-12 16:12:54,574 INFO reduce.MergeManagerImpl (MergeManagerImpl.java:finalMerge(673)) - finalMerge called with 2 in-memory map-outputs and 0 on-disk map-outputs
2015-02-12 16:12:54,575 INFO mapred.Merger (Merger.java:merge(606)) - Merging 2 sorted segments
2015-02-12 16:12:54,576 INFO mapred.Merger (Merger.java:merge(705)) - Down to the last merge-pass, with 2 segments left of total size: 156 bytes
2015-02-12 16:12:54,577 INFO reduce.MergeManagerImpl (MergeManagerImpl.java:finalMerge(750)) - Merged 2 segments, 174 bytes to disk to satisfy reduce memory limit
2015-02-12 16:12:54,577 INFO reduce.MergeManagerImpl (MergeManagerImpl.java:finalMerge(780)) - Merging 1 files, 176 bytes from disk
2015-02-12 16:12:54,577 INFO reduce.MergeManagerImpl (MergeManagerImpl.java:finalMerge(795)) - Merging 0 segments, 0 bytes from memory into reduce
2015-02-12 16:12:54,577 INFO mapred.Merger (Merger.java:merge(606)) - Merging 1 sorted segments
2015-02-12 16:12:54,578 INFO mapred.Merger (Merger.java:merge(705)) - Down to the last merge-pass, with 1 segments left of total size: 163 bytes
2015-02-12 16:12:54,578 INFO mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) - 2 / 2 copied.
2015-02-12 16:12:54,578 INFO hadoop.FlowReducer (FlowReducer.java:configure(78)) - cascading version: 2.6.3
2015-02-12 16:12:54,578 INFO hadoop.FlowReducer (FlowReducer.java:configure(79)) - child jvm opts: -Xmx200m
2015-02-12 16:12:54,581 INFO hadoop.FlowReducer (FlowReducer.java:configure(97)) - sourcing from: GroupBy(lhs+rhs)[by: lhs:[{1}:'num'] rhs:[{1}:'num']]
2015-02-12 16:12:54,582 INFO hadoop.FlowReducer (FlowReducer.java:configure(100)) - sinking to: Hfs["TextLine[['item']->['num', 'char']]"]["/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/test/output/hadoop2-mr1/assemblyhelpersplatform/uniquemerge-nondeterministic"]
2015-02-12 16:12:54,590 INFO mapred.Task (Task.java:done(1001)) - Task:attempt_local1683387041_0037_r_000000_0 is done. And is in the process of committing
2015-02-12 16:12:54,591 INFO mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) - 2 / 2 copied.
2015-02-12 16:12:54,591 INFO mapred.Task (Task.java:commit(1162)) - Task attempt_local1683387041_0037_r_000000_0 is allowed to commit now
2015-02-12 16:12:54,592 INFO output.FileOutputCommitter (FileOutputCommitter.java:commitTask(439)) - Saved output of task 'attempt_local1683387041_0037_r_000000_0' to file:/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/test/output/hadoop2-mr1/assemblyhelpersplatform/uniquemerge-nondeterministic/_temporary/0/task_local1683387041_0037_r_000000
2015-02-12 16:12:54,593 INFO mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) - reduce > reduce
2015-02-12 16:12:54,593 INFO mapred.Task (Task.java:sendDone(1121)) - Task 'attempt_local1683387041_0037_r_000000_0' done.
2015-02-12 16:12:54,593 INFO mapred.LocalJobRunner (LocalJobRunner.java:run(325)) - Finishing task: attempt_local1683387041_0037_r_000000_0
2015-02-12 16:12:54,593 INFO mapred.LocalJobRunner (LocalJobRunner.java:runTasks(456)) - reduce task executor complete.
2015-02-12 16:12:54,616 INFO util.Hadoop18TapUtil (Hadoop18TapUtil.java:cleanTempPath(219)) - deleting temp path /Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/test/output/hadoop2-mr1/assemblyhelpersplatform/uniquemerge-nondeterministic/_temporary
2015-02-12 16:12:54,636 INFO mapred.FileInputFormat (FileInputFormat.java:listStatus(247)) - Total input paths to process : 1
2015-02-12 16:12:54,637 INFO cascading.PlatformTestCase (PlatformTestCase.java:tearDown(209)) - copying to local /Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/test/output//hadoop2-mr1/assemblyhelpersplatform/uniquemerge-nondeterministic
2015-02-12 16:12:54,647 INFO util.HadoopUtil (HadoopUtil.java:findMainClass(336)) - using default application jar, may cause class not found exceptions on the cluster
2015-02-12 16:12:54,647 INFO planner.HadoopPlanner (HadoopPlanner.java:initialize(225)) - using application jar: /Users/soberoi/.gradle/caches/modules-2/files-2.1/cascading/cascading-hadoop2-mr1/2.6.3/9ae016e2b8d246b9c1de9168e715d9f1b971aa4a/cascading-hadoop2-mr1-2.6.3.jar
2015-02-12 16:12:54,651 INFO Configuration.deprecation (Configuration.java:warnOnceIfDeprecated(1022)) - mapred.used.genericoptionsparser is deprecated. Instead, use mapreduce.client.genericoptionsparser.used
2015-02-12 16:12:54,652 INFO Configuration.deprecation (Configuration.java:warnOnceIfDeprecated(1022)) - mapred.job.tracker is deprecated. Instead, use mapreduce.jobtracker.address
2015-02-12 16:12:54,652 INFO Configuration.deprecation (Configuration.java:warnOnceIfDeprecated(1022)) - mapred.output.compress is deprecated. Instead, use mapreduce.output.fileoutputformat.compress
2015-02-12 16:12:54,655 INFO flow.Flow (BaseFlow.java:logInfo(1378)) - [shape] starting
2015-02-12 16:12:54,655 INFO flow.Flow (BaseFlow.java:logInfo(1378)) - [shape] source: Hfs["TextLine[['offset', 'line']->[ALL]]"]["/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/classes/test/data/lhs.txt"]
2015-02-12 16:12:54,656 INFO flow.Flow (BaseFlow.java:logInfo(1378)) - [shape] sink: Hfs["TextLine[['item']->['num', 'char']]"]["/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/test/output/hadoop2-mr1/assemblyhelpersplatform/unique"]
2015-02-12 16:12:54,656 INFO flow.Flow (BaseFlow.java:logInfo(1378)) - [shape] parallel execution is enabled: false
2015-02-12 16:12:54,656 INFO flow.Flow (BaseFlow.java:logInfo(1378)) - [shape] starting jobs: 1
2015-02-12 16:12:54,656 INFO flow.Flow (BaseFlow.java:logInfo(1378)) - [shape] allocating threads: 1
2015-02-12 16:12:54,656 INFO flow.FlowStep (BaseFlowStep.java:logInfo(834)) - [shape] starting step: (1/1) ...blyhelpersplatform/unique
2015-02-12 16:12:54,657 INFO jvm.JvmMetrics (JvmMetrics.java:init(71)) - Cannot initialize JVM Metrics with processName=JobTracker, sessionId= - already initialized
2015-02-12 16:12:54,657 INFO jvm.JvmMetrics (JvmMetrics.java:init(71)) - Cannot initialize JVM Metrics with processName=JobTracker, sessionId= - already initialized
2015-02-12 16:12:54,678 INFO mapred.FileInputFormat (FileInputFormat.java:listStatus(247)) - Total input paths to process : 1
2015-02-12 16:12:54,693 INFO mapreduce.JobSubmitter (JobSubmitter.java:submitJobInternal(492)) - number of splits:1
2015-02-12 16:12:54,704 INFO mapreduce.JobSubmitter (JobSubmitter.java:printTokens(575)) - Submitting tokens for job: job_local2117717862_0038
2015-02-12 16:12:54,713 WARN conf.Configuration (Configuration.java:loadProperty(2510)) - file:/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/tmp/cascading/staging/soberoi2117717862/.staging/job_local2117717862_0038/job.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.retry.interval; Ignoring.
2015-02-12 16:12:54,714 WARN conf.Configuration (Configuration.java:loadProperty(2510)) - file:/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/tmp/cascading/staging/soberoi2117717862/.staging/job_local2117717862_0038/job.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.attempts; Ignoring.
2015-02-12 16:12:54,741 WARN conf.Configuration (Configuration.java:loadProperty(2510)) - file:/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/test/tmp/mapred/local/localRunner/soberoi/job_local2117717862_0038/job_local2117717862_0038.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.retry.interval; Ignoring.
2015-02-12 16:12:54,742 WARN conf.Configuration (Configuration.java:loadProperty(2510)) - file:/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/test/tmp/mapred/local/localRunner/soberoi/job_local2117717862_0038/job_local2117717862_0038.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.attempts; Ignoring.
2015-02-12 16:12:54,742 INFO mapreduce.Job (Job.java:submit(1299)) - The url to track the job: http://localhost:8080/
2015-02-12 16:12:54,743 INFO mapred.LocalJobRunner (LocalJobRunner.java:createOutputCommitter(471)) - OutputCommitter set in config null
2015-02-12 16:12:54,743 INFO flow.FlowStep (BaseFlowStep.java:logInfo(834)) - [shape] submitted hadoop job: job_local2117717862_0038
2015-02-12 16:12:54,743 INFO mapred.LocalJobRunner (LocalJobRunner.java:createOutputCommitter(489)) - OutputCommitter is org.apache.hadoop.mapred.FileOutputCommitter
2015-02-12 16:12:54,743 INFO flow.FlowStep (BaseFlowStep.java:logInfo(834)) - [shape] tracking url: http://localhost:8080/
2015-02-12 16:12:54,744 INFO mapred.LocalJobRunner (LocalJobRunner.java:runTasks(448)) - Waiting for map tasks
2015-02-12 16:12:54,744 INFO mapred.LocalJobRunner (LocalJobRunner.java:run(224)) - Starting task: attempt_local2117717862_0038_m_000000_0
2015-02-12 16:12:54,745 INFO util.ProcfsBasedProcessTree (ProcfsBasedProcessTree.java:isAvailable(181)) - ProcfsBasedProcessTree currently is supported only on Linux.
2015-02-12 16:12:54,745 INFO mapred.Task (Task.java:initialize(587)) - Using ResourceCalculatorProcessTree : null
2015-02-12 16:12:54,746 INFO io.MultiInputSplit (MultiInputSplit.java:readFields(161)) - current split input path: file:/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/classes/test/data/lhs.txt
2015-02-12 16:12:54,746 INFO mapred.MapTask (MapTask.java:updateJobWithSplit(482)) - Processing split: cascading.tap.hadoop.io.MultiInputSplit@2a0f331b
2015-02-12 16:12:54,747 INFO mapred.MapTask (MapTask.java:runOldMapper(436)) - numReduceTasks: 1
2015-02-12 16:12:54,763 INFO mapred.MapTask (MapTask.java:setEquator(1202)) - (EQUATOR) 0 kvi 26214396(104857584)
2015-02-12 16:12:54,763 INFO mapred.MapTask (MapTask.java:init(995)) - mapreduce.task.io.sort.mb: 100
2015-02-12 16:12:54,763 INFO mapred.MapTask (MapTask.java:init(996)) - soft limit at 83886080
2015-02-12 16:12:54,763 INFO mapred.MapTask (MapTask.java:init(997)) - bufstart = 0; bufvoid = 104857600
2015-02-12 16:12:54,763 INFO mapred.MapTask (MapTask.java:init(998)) - kvstart = 26214396; length = 6553600
2015-02-12 16:12:54,764 INFO mapred.MapTask (MapTask.java:createSortingCollector(402)) - Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer
2015-02-12 16:12:54,764 INFO hadoop.FlowMapper (FlowMapper.java:configure(71)) - cascading version: 2.6.3
2015-02-12 16:12:54,764 INFO hadoop.FlowMapper (FlowMapper.java:configure(72)) - child jvm opts: -Xmx200m
2015-02-12 16:12:54,766 INFO hadoop.FlowMapper (FlowMapper.java:configure(87)) - sourcing from: Hfs["TextLine[['offset', 'line']->[ALL]]"]["/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/classes/test/data/lhs.txt"]
2015-02-12 16:12:54,766 INFO hadoop.FlowMapper (FlowMapper.java:configure(90)) - sinking to: GroupBy(shape)[by:[{1}:'num']]
2015-02-12 16:12:54,768 INFO mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) -
2015-02-12 16:12:54,768 INFO mapred.MapTask (MapTask.java:flush(1457)) - Starting flush of map output
2015-02-12 16:12:54,768 INFO mapred.MapTask (MapTask.java:flush(1475)) - Spilling map output
2015-02-12 16:12:54,768 INFO mapred.MapTask (MapTask.java:flush(1476)) - bufstart = 0; bufend = 75; bufvoid = 104857600
2015-02-12 16:12:54,768 INFO mapred.MapTask (MapTask.java:flush(1478)) - kvstart = 26214396(104857584); kvend = 26214380(104857520); length = 17/6553600
2015-02-12 16:12:54,770 INFO mapred.MapTask (MapTask.java:sortAndSpill(1660)) - Finished spill 0
2015-02-12 16:12:54,772 INFO mapred.Task (Task.java:done(1001)) - Task:attempt_local2117717862_0038_m_000000_0 is done. And is in the process of committing
2015-02-12 16:12:54,773 INFO mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) - file:/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/classes/test/data/lhs.txt:0+52
2015-02-12 16:12:54,773 INFO mapred.Task (Task.java:sendDone(1121)) - Task 'attempt_local2117717862_0038_m_000000_0' done.
2015-02-12 16:12:54,773 INFO mapred.LocalJobRunner (LocalJobRunner.java:run(249)) - Finishing task: attempt_local2117717862_0038_m_000000_0
2015-02-12 16:12:54,774 INFO mapred.LocalJobRunner (LocalJobRunner.java:runTasks(456)) - map task executor complete.
2015-02-12 16:12:54,774 INFO mapred.LocalJobRunner (LocalJobRunner.java:runTasks(448)) - Waiting for reduce tasks
2015-02-12 16:12:54,774 INFO mapred.LocalJobRunner (LocalJobRunner.java:run(302)) - Starting task: attempt_local2117717862_0038_r_000000_0
2015-02-12 16:12:54,775 INFO util.ProcfsBasedProcessTree (ProcfsBasedProcessTree.java:isAvailable(181)) - ProcfsBasedProcessTree currently is supported only on Linux.
2015-02-12 16:12:54,775 INFO mapred.Task (Task.java:initialize(587)) - Using ResourceCalculatorProcessTree : null
2015-02-12 16:12:54,775 INFO mapred.ReduceTask (ReduceTask.java:run(362)) - Using ShuffleConsumerPlugin: org.apache.hadoop.mapreduce.task.reduce.Shuffle@1315391f
2015-02-12 16:12:54,776 INFO reduce.MergeManagerImpl (MergeManagerImpl.java:<init>(195)) - MergerManager: memoryLimit=493250144, maxSingleShuffleLimit=123312536, mergeThreshold=325545120, ioSortFactor=10, memToMemMergeOutputsThreshold=10
2015-02-12 16:12:54,777 INFO reduce.EventFetcher (EventFetcher.java:run(61)) - attempt_local2117717862_0038_r_000000_0 Thread started: EventFetcher for fetching Map Completion Events
2015-02-12 16:12:54,777 INFO reduce.LocalFetcher (LocalFetcher.java:copyMapOutput(141)) - localfetcher#31 about to shuffle output of map attempt_local2117717862_0038_m_000000_0 decomp: 87 len: 91 to MEMORY
2015-02-12 16:12:54,778 INFO reduce.InMemoryMapOutput (InMemoryMapOutput.java:shuffle(100)) - Read 87 bytes from map-output for attempt_local2117717862_0038_m_000000_0
2015-02-12 16:12:54,778 INFO reduce.MergeManagerImpl (MergeManagerImpl.java:closeInMemoryFile(313)) - closeInMemoryFile -> map-output of size: 87, inMemoryMapOutputs.size() -> 1, commitMemory -> 0, usedMemory ->87
2015-02-12 16:12:54,778 INFO reduce.EventFetcher (EventFetcher.java:run(76)) - EventFetcher is interrupted.. Returning
2015-02-12 16:12:54,779 INFO mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) - 1 / 1 copied.
2015-02-12 16:12:54,779 INFO reduce.MergeManagerImpl (MergeManagerImpl.java:finalMerge(673)) - finalMerge called with 1 in-memory map-outputs and 0 on-disk map-outputs
2015-02-12 16:12:54,780 INFO mapred.Merger (Merger.java:merge(606)) - Merging 1 sorted segments
2015-02-12 16:12:54,780 INFO mapred.Merger (Merger.java:merge(705)) - Down to the last merge-pass, with 1 segments left of total size: 78 bytes
2015-02-12 16:12:54,781 INFO reduce.MergeManagerImpl (MergeManagerImpl.java:finalMerge(750)) - Merged 1 segments, 87 bytes to disk to satisfy reduce memory limit
2015-02-12 16:12:54,781 INFO reduce.MergeManagerImpl (MergeManagerImpl.java:finalMerge(780)) - Merging 1 files, 91 bytes from disk
2015-02-12 16:12:54,782 INFO reduce.MergeManagerImpl (MergeManagerImpl.java:finalMerge(795)) - Merging 0 segments, 0 bytes from memory into reduce
2015-02-12 16:12:54,782 INFO mapred.Merger (Merger.java:merge(606)) - Merging 1 sorted segments
2015-02-12 16:12:54,782 INFO mapred.Merger (Merger.java:merge(705)) - Down to the last merge-pass, with 1 segments left of total size: 78 bytes
2015-02-12 16:12:54,782 INFO mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) - 1 / 1 copied.
2015-02-12 16:12:54,782 INFO hadoop.FlowReducer (FlowReducer.java:configure(78)) - cascading version: 2.6.3
2015-02-12 16:12:54,783 INFO hadoop.FlowReducer (FlowReducer.java:configure(79)) - child jvm opts: -Xmx200m
2015-02-12 16:12:54,785 INFO hadoop.FlowReducer (FlowReducer.java:configure(97)) - sourcing from: GroupBy(shape)[by:[{1}:'num']]
2015-02-12 16:12:54,785 INFO hadoop.FlowReducer (FlowReducer.java:configure(100)) - sinking to: Hfs["TextLine[['item']->['num', 'char']]"]["/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/test/output/hadoop2-mr1/assemblyhelpersplatform/unique"]
2015-02-12 16:12:54,791 INFO mapred.Task (Task.java:done(1001)) - Task:attempt_local2117717862_0038_r_000000_0 is done. And is in the process of committing
2015-02-12 16:12:54,792 INFO mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) - 1 / 1 copied.
2015-02-12 16:12:54,792 INFO mapred.Task (Task.java:commit(1162)) - Task attempt_local2117717862_0038_r_000000_0 is allowed to commit now
2015-02-12 16:12:54,793 INFO output.FileOutputCommitter (FileOutputCommitter.java:commitTask(439)) - Saved output of task 'attempt_local2117717862_0038_r_000000_0' to file:/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/test/output/hadoop2-mr1/assemblyhelpersplatform/unique/_temporary/0/task_local2117717862_0038_r_000000
2015-02-12 16:12:54,793 INFO mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) - reduce > reduce
2015-02-12 16:12:54,793 INFO mapred.Task (Task.java:sendDone(1121)) - Task 'attempt_local2117717862_0038_r_000000_0' done.
2015-02-12 16:12:54,794 INFO mapred.LocalJobRunner (LocalJobRunner.java:run(325)) - Finishing task: attempt_local2117717862_0038_r_000000_0
2015-02-12 16:12:54,794 INFO mapred.LocalJobRunner (LocalJobRunner.java:runTasks(456)) - reduce task executor complete.
2015-02-12 16:12:54,804 INFO util.Hadoop18TapUtil (Hadoop18TapUtil.java:cleanTempPath(219)) - deleting temp path /Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/test/output/hadoop2-mr1/assemblyhelpersplatform/unique/_temporary
2015-02-12 16:12:54,824 INFO mapred.FileInputFormat (FileInputFormat.java:listStatus(247)) - Total input paths to process : 1
2015-02-12 16:12:54,825 INFO cascading.PlatformTestCase (PlatformTestCase.java:tearDown(209)) - copying to local /Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/test/output//hadoop2-mr1/assemblyhelpersplatform/unique
2015-02-12 16:12:54,837 INFO util.HadoopUtil (HadoopUtil.java:findMainClass(336)) - using default application jar, may cause class not found exceptions on the cluster
2015-02-12 16:12:54,837 INFO planner.HadoopPlanner (HadoopPlanner.java:initialize(225)) - using application jar: /Users/soberoi/.gradle/caches/modules-2/files-2.1/cascading/cascading-hadoop2-mr1/2.6.3/9ae016e2b8d246b9c1de9168e715d9f1b971aa4a/cascading-hadoop2-mr1-2.6.3.jar
2015-02-12 16:12:54,840 INFO Configuration.deprecation (Configuration.java:warnOnceIfDeprecated(1022)) - mapred.used.genericoptionsparser is deprecated. Instead, use mapreduce.client.genericoptionsparser.used
2015-02-12 16:12:54,841 INFO Configuration.deprecation (Configuration.java:warnOnceIfDeprecated(1022)) - mapred.job.tracker is deprecated. Instead, use mapreduce.jobtracker.address
2015-02-12 16:12:54,841 INFO Configuration.deprecation (Configuration.java:warnOnceIfDeprecated(1022)) - mapred.output.compress is deprecated. Instead, use mapreduce.output.fileoutputformat.compress
2015-02-12 16:12:54,845 INFO flow.Flow (BaseFlow.java:logInfo(1378)) - [count] starting
2015-02-12 16:12:54,845 INFO flow.Flow (BaseFlow.java:logInfo(1378)) - [count] source: Hfs["TextDelimited[['num', 'char']]"]["/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/classes/test/data/lhs.txt"]
2015-02-12 16:12:54,845 INFO flow.Flow (BaseFlow.java:logInfo(1378)) - [count] sink: Hfs["TextDelimited[['notnull', 'null']]"]["/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/test/output/hadoop2-mr1/assemblyhelpersplatform/countnullnotnull"]
2015-02-12 16:12:54,845 INFO flow.Flow (BaseFlow.java:logInfo(1378)) - [count] parallel execution is enabled: false
2015-02-12 16:12:54,845 INFO flow.Flow (BaseFlow.java:logInfo(1378)) - [count] starting jobs: 1
2015-02-12 16:12:54,845 INFO flow.Flow (BaseFlow.java:logInfo(1378)) - [count] allocating threads: 1
2015-02-12 16:12:54,846 INFO flow.FlowStep (BaseFlowStep.java:logInfo(834)) - [count] starting step: (1/1) ...platform/countnullnotnull
2015-02-12 16:12:54,846 INFO jvm.JvmMetrics (JvmMetrics.java:init(71)) - Cannot initialize JVM Metrics with processName=JobTracker, sessionId= - already initialized
2015-02-12 16:12:54,847 INFO jvm.JvmMetrics (JvmMetrics.java:init(71)) - Cannot initialize JVM Metrics with processName=JobTracker, sessionId= - already initialized
2015-02-12 16:12:54,865 INFO mapred.FileInputFormat (FileInputFormat.java:listStatus(247)) - Total input paths to process : 1
2015-02-12 16:12:54,881 INFO mapreduce.JobSubmitter (JobSubmitter.java:submitJobInternal(492)) - number of splits:1
2015-02-12 16:12:54,892 INFO mapreduce.JobSubmitter (JobSubmitter.java:printTokens(575)) - Submitting tokens for job: job_local1121988028_0039
2015-02-12 16:12:54,901 WARN conf.Configuration (Configuration.java:loadProperty(2510)) - file:/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/tmp/cascading/staging/soberoi1121988028/.staging/job_local1121988028_0039/job.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.retry.interval; Ignoring.
2015-02-12 16:12:54,903 WARN conf.Configuration (Configuration.java:loadProperty(2510)) - file:/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/tmp/cascading/staging/soberoi1121988028/.staging/job_local1121988028_0039/job.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.attempts; Ignoring.
2015-02-12 16:12:54,934 WARN conf.Configuration (Configuration.java:loadProperty(2510)) - file:/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/test/tmp/mapred/local/localRunner/soberoi/job_local1121988028_0039/job_local1121988028_0039.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.retry.interval; Ignoring.
2015-02-12 16:12:54,935 WARN conf.Configuration (Configuration.java:loadProperty(2510)) - file:/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/test/tmp/mapred/local/localRunner/soberoi/job_local1121988028_0039/job_local1121988028_0039.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.attempts; Ignoring.
2015-02-12 16:12:54,935 INFO mapreduce.Job (Job.java:submit(1299)) - The url to track the job: http://localhost:8080/
2015-02-12 16:12:54,935 INFO flow.FlowStep (BaseFlowStep.java:logInfo(834)) - [count] submitted hadoop job: job_local1121988028_0039
2015-02-12 16:12:54,935 INFO mapred.LocalJobRunner (LocalJobRunner.java:createOutputCommitter(471)) - OutputCommitter set in config null
2015-02-12 16:12:54,935 INFO flow.FlowStep (BaseFlowStep.java:logInfo(834)) - [count] tracking url: http://localhost:8080/
2015-02-12 16:12:54,936 INFO mapred.LocalJobRunner (LocalJobRunner.java:createOutputCommitter(489)) - OutputCommitter is org.apache.hadoop.mapred.FileOutputCommitter
2015-02-12 16:12:54,937 INFO mapred.LocalJobRunner (LocalJobRunner.java:runTasks(448)) - Waiting for map tasks
2015-02-12 16:12:54,937 INFO mapred.LocalJobRunner (LocalJobRunner.java:run(224)) - Starting task: attempt_local1121988028_0039_m_000000_0
2015-02-12 16:12:54,938 INFO util.ProcfsBasedProcessTree (ProcfsBasedProcessTree.java:isAvailable(181)) - ProcfsBasedProcessTree currently is supported only on Linux.
2015-02-12 16:12:54,938 INFO mapred.Task (Task.java:initialize(587)) - Using ResourceCalculatorProcessTree : null
2015-02-12 16:12:54,939 INFO io.MultiInputSplit (MultiInputSplit.java:readFields(161)) - current split input path: file:/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/classes/test/data/lhs.txt
2015-02-12 16:12:54,939 INFO mapred.MapTask (MapTask.java:updateJobWithSplit(482)) - Processing split: cascading.tap.hadoop.io.MultiInputSplit@63b30992
2015-02-12 16:12:54,939 INFO mapred.MapTask (MapTask.java:runOldMapper(436)) - numReduceTasks: 1
2015-02-12 16:12:54,955 INFO mapred.MapTask (MapTask.java:setEquator(1202)) - (EQUATOR) 0 kvi 26214396(104857584)
2015-02-12 16:12:54,955 INFO mapred.MapTask (MapTask.java:init(995)) - mapreduce.task.io.sort.mb: 100
2015-02-12 16:12:54,955 INFO mapred.MapTask (MapTask.java:init(996)) - soft limit at 83886080
2015-02-12 16:12:54,955 INFO mapred.MapTask (MapTask.java:init(997)) - bufstart = 0; bufvoid = 104857600
2015-02-12 16:12:54,955 INFO mapred.MapTask (MapTask.java:init(998)) - kvstart = 26214396; length = 6553600
2015-02-12 16:12:54,955 INFO mapred.MapTask (MapTask.java:createSortingCollector(402)) - Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer
2015-02-12 16:12:54,956 INFO hadoop.FlowMapper (FlowMapper.java:configure(71)) - cascading version: 2.6.3
2015-02-12 16:12:54,956 INFO hadoop.FlowMapper (FlowMapper.java:configure(72)) - child jvm opts: -Xmx200m
2015-02-12 16:12:54,959 INFO hadoop.FlowMapper (FlowMapper.java:configure(87)) - sourcing from: Hfs["TextDelimited[['num', 'char']]"]["/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/classes/test/data/lhs.txt"]
2015-02-12 16:12:54,959 INFO hadoop.FlowMapper (FlowMapper.java:configure(90)) - sinking to: GroupBy(count)[by:[{?}:NONE]]
2015-02-12 16:12:54,959 INFO assembly.AggregateBy (AggregateBy.java:prepare(267)) - using threshold value: 2
2015-02-12 16:12:54,962 INFO mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) -
2015-02-12 16:12:54,962 INFO mapred.MapTask (MapTask.java:flush(1457)) - Starting flush of map output
2015-02-12 16:12:54,962 INFO mapred.MapTask (MapTask.java:flush(1475)) - Spilling map output
2015-02-12 16:12:54,963 INFO mapred.MapTask (MapTask.java:flush(1476)) - bufstart = 0; bufend = 6; bufvoid = 104857600
2015-02-12 16:12:54,963 INFO mapred.MapTask (MapTask.java:flush(1478)) - kvstart = 26214396(104857584); kvend = 26214396(104857584); length = 1/6553600
2015-02-12 16:12:54,964 INFO mapred.MapTask (MapTask.java:sortAndSpill(1660)) - Finished spill 0
2015-02-12 16:12:54,965 INFO mapred.Task (Task.java:done(1001)) - Task:attempt_local1121988028_0039_m_000000_0 is done. And is in the process of committing
2015-02-12 16:12:54,966 INFO mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) - file:/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/classes/test/data/lhs.txt:0+52
2015-02-12 16:12:54,966 INFO mapred.Task (Task.java:sendDone(1121)) - Task 'attempt_local1121988028_0039_m_000000_0' done.
2015-02-12 16:12:54,966 INFO mapred.LocalJobRunner (LocalJobRunner.java:run(249)) - Finishing task: attempt_local1121988028_0039_m_000000_0
2015-02-12 16:12:54,966 INFO mapred.LocalJobRunner (LocalJobRunner.java:runTasks(456)) - map task executor complete.
2015-02-12 16:12:54,967 INFO mapred.LocalJobRunner (LocalJobRunner.java:runTasks(448)) - Waiting for reduce tasks
2015-02-12 16:12:54,967 INFO mapred.LocalJobRunner (LocalJobRunner.java:run(302)) - Starting task: attempt_local1121988028_0039_r_000000_0
2015-02-12 16:12:54,968 INFO util.ProcfsBasedProcessTree (ProcfsBasedProcessTree.java:isAvailable(181)) - ProcfsBasedProcessTree currently is supported only on Linux.
2015-02-12 16:12:54,968 INFO mapred.Task (Task.java:initialize(587)) - Using ResourceCalculatorProcessTree : null
2015-02-12 16:12:54,968 INFO mapred.ReduceTask (ReduceTask.java:run(362)) - Using ShuffleConsumerPlugin: org.apache.hadoop.mapreduce.task.reduce.Shuffle@6db52762
2015-02-12 16:12:54,968 INFO reduce.MergeManagerImpl (MergeManagerImpl.java:<init>(195)) - MergerManager: memoryLimit=493250144, maxSingleShuffleLimit=123312536, mergeThreshold=325545120, ioSortFactor=10, memToMemMergeOutputsThreshold=10
2015-02-12 16:12:54,969 INFO reduce.EventFetcher (EventFetcher.java:run(61)) - attempt_local1121988028_0039_r_000000_0 Thread started: EventFetcher for fetching Map Completion Events
2015-02-12 16:12:54,969 INFO reduce.LocalFetcher (LocalFetcher.java:copyMapOutput(141)) - localfetcher#32 about to shuffle output of map attempt_local1121988028_0039_m_000000_0 decomp: 10 len: 14 to MEMORY
2015-02-12 16:12:54,970 INFO reduce.InMemoryMapOutput (InMemoryMapOutput.java:shuffle(100)) - Read 10 bytes from map-output for attempt_local1121988028_0039_m_000000_0
2015-02-12 16:12:54,970 INFO reduce.MergeManagerImpl (MergeManagerImpl.java:closeInMemoryFile(313)) - closeInMemoryFile -> map-output of size: 10, inMemoryMapOutputs.size() -> 1, commitMemory -> 0, usedMemory ->10
2015-02-12 16:12:54,970 INFO reduce.EventFetcher (EventFetcher.java:run(76)) - EventFetcher is interrupted.. Returning
2015-02-12 16:12:54,971 INFO mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) - 1 / 1 copied.
2015-02-12 16:12:54,971 INFO reduce.MergeManagerImpl (MergeManagerImpl.java:finalMerge(673)) - finalMerge called with 1 in-memory map-outputs and 0 on-disk map-outputs
2015-02-12 16:12:54,972 INFO mapred.Merger (Merger.java:merge(606)) - Merging 1 sorted segments
2015-02-12 16:12:54,972 INFO mapred.Merger (Merger.java:merge(705)) - Down to the last merge-pass, with 1 segments left of total size: 7 bytes
2015-02-12 16:12:54,973 INFO reduce.MergeManagerImpl (MergeManagerImpl.java:finalMerge(750)) - Merged 1 segments, 10 bytes to disk to satisfy reduce memory limit
2015-02-12 16:12:54,973 INFO reduce.MergeManagerImpl (MergeManagerImpl.java:finalMerge(780)) - Merging 1 files, 14 bytes from disk
2015-02-12 16:12:54,973 INFO reduce.MergeManagerImpl (MergeManagerImpl.java:finalMerge(795)) - Merging 0 segments, 0 bytes from memory into reduce
2015-02-12 16:12:54,973 INFO mapred.Merger (Merger.java:merge(606)) - Merging 1 sorted segments
2015-02-12 16:12:54,973 INFO mapred.Merger (Merger.java:merge(705)) - Down to the last merge-pass, with 1 segments left of total size: 7 bytes
2015-02-12 16:12:54,974 INFO mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) - 1 / 1 copied.
2015-02-12 16:12:54,974 INFO hadoop.FlowReducer (FlowReducer.java:configure(78)) - cascading version: 2.6.3
2015-02-12 16:12:54,974 INFO hadoop.FlowReducer (FlowReducer.java:configure(79)) - child jvm opts: -Xmx200m
2015-02-12 16:12:54,978 INFO hadoop.FlowReducer (FlowReducer.java:configure(97)) - sourcing from: GroupBy(count)[by:[{?}:NONE]]
2015-02-12 16:12:54,978 INFO hadoop.FlowReducer (FlowReducer.java:configure(100)) - sinking to: Hfs["TextDelimited[['notnull', 'null']]"]["/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/test/output/hadoop2-mr1/assemblyhelpersplatform/countnullnotnull"]
2015-02-12 16:12:54,984 INFO mapred.Task (Task.java:done(1001)) - Task:attempt_local1121988028_0039_r_000000_0 is done. And is in the process of committing
2015-02-12 16:12:54,985 INFO mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) - 1 / 1 copied.
2015-02-12 16:12:54,986 INFO mapred.Task (Task.java:commit(1162)) - Task attempt_local1121988028_0039_r_000000_0 is allowed to commit now
2015-02-12 16:12:54,986 INFO output.FileOutputCommitter (FileOutputCommitter.java:commitTask(439)) - Saved output of task 'attempt_local1121988028_0039_r_000000_0' to file:/Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/test/output/hadoop2-mr1/assemblyhelpersplatform/countnullnotnull/_temporary/0/task_local1121988028_0039_r_000000
2015-02-12 16:12:54,987 INFO mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) - reduce > reduce
2015-02-12 16:12:54,987 INFO mapred.Task (Task.java:sendDone(1121)) - Task 'attempt_local1121988028_0039_r_000000_0' done.
2015-02-12 16:12:54,987 INFO mapred.LocalJobRunner (LocalJobRunner.java:run(325)) - Finishing task: attempt_local1121988028_0039_r_000000_0
2015-02-12 16:12:54,987 INFO mapred.LocalJobRunner (LocalJobRunner.java:runTasks(456)) - reduce task executor complete.
2015-02-12 16:12:55,006 INFO util.Hadoop18TapUtil (Hadoop18TapUtil.java:cleanTempPath(219)) - deleting temp path /Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/test/output/hadoop2-mr1/assemblyhelpersplatform/countnullnotnull/_temporary
2015-02-12 16:12:55,026 INFO mapred.FileInputFormat (FileInputFormat.java:listStatus(247)) - Total input paths to process : 1
2015-02-12 16:12:55,046 INFO mapred.FileInputFormat (FileInputFormat.java:listStatus(247)) - Total input paths to process : 1
2015-02-12 16:12:55,047 INFO cascading.PlatformTestCase (PlatformTestCase.java:tearDown(209)) - copying to local /Users/soberoi/src/compatibility/cascading.compatibility/cloudera/build/test/output//hadoop2-mr1/assemblyhelpersplatform/countnullnotnull