Spark经典案例5-求最大最小值

2024-06-28 16:01:33

字体：大中小

来源：转载

供稿：网友

数据准备 eightteen_a.txt 102 10 39 109 200 11 3 90 28

eightteen_b.txt 5 2 30 838 10005

package ClassicCaseimport org.apache.spark.{SparkConf, SparkContext}/** * 业务场景：求最大最小值 * Created by YJ on 2017/2/8. */object case5 { def main(args: Array[String]): Unit = { val conf = new SparkConf().setMaster("local").setAppName("reduce") val sc = new SparkContext(conf) sc.setLogLevel("ERROR") val fifth = sc.textFile("hdfs://192.168.109.130:8020//user/flume/ClassicCase/case5/*", 2) val res = fifth.filter(_.trim().length>0).map(line => ("key",line.trim.toInt)).groupByKey().map(x => { var min = Integer.MAX_VALUE var max = Integer.MIN_VALUE for(num <- x._2){ if(num>max){ max = num } if(num<min){ min = num } } (max,min) }).collect.foreach(x => { PRintln("max/t"+x._1) println("min/t"+x._2) }) }}

结果输出 max 10005 min 2

上一篇：hadoop源码查看

下一篇：Zeppelin0.5.6使用spark解释器