spark + python安装教程
spark + python安装教程

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
# coding=UTF-8

import os

os.environ['JAVA_HOME'] = r'D:\software\spark\java8'  # 这个路径更换为你自己的java安装目录
import findspark

findspark.init()
from pyspark import SparkContext


def show(x):
    print(x)


sc = SparkContext("local", "First App")
lines = sc.textFile("./test/word.txt").cache()
words = lines.flatMap(lambda line: line.split(" "), True)
pairWords = words.map(lambda word: (word, 1), True)
result = pairWords.reduceByKey(lambda v1, v2: v1 + v2, 3)
result.foreach(lambda x: show(x))
# result.saveAsTextFile("./wc-result2")

相关阅读

🎓 AI 编程实战课程

程序员晚枫专注AI编程培训，通过 《30讲 · AI编程训练营》，让小白也能用AI做出实际项目。帮你从零上手！