FilesExpand file tree

 master

/

coin.py

Copy path

Blame

More file actions

Blame

More file actions

Latest commit

 

History

History

History

37 lines (25 loc) · 775 Bytes

 master

/

coin.py

Copy path

Top

File metadata and controls

• Code
• Blame

37 lines (25 loc) · 775 Bytes

Raw

Copy raw file

Download raw file

Open symbols panel

Edit and raw actions

1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

16

17

18

19

20

21

22

23

24

25

26

27

28

29

30

31

32

33

34

35

36

37

'''

1.9: Introduction to RDDs: Functions, Transformations, and Actions

Counting coin flips example with SparkContext

Run with: python coin.py

'''

import pyspark as ps

import random

# initialize SparkContext

sc = ps.SparkContext()

# Turn off debug logging to console

logger = sc._jvm.org.apache.log4j

logger.LogManager.getLogger("org"). setLevel( logger.Level.ERROR )

logger.LogManager.getLogger("akka").setLevel( logger.Level.ERROR )

###################################

flips = 1000000

# lazy eval

coins = xrange(flips)

# lazy eval, nothing executed

heads = sc.parallelize(coins) \

.map(lambda i: random.random()) \

.filter(lambda r: r < 0.51) \

.count()

print "{0} heads flipped!".format(heads)

# shutdown SparkContext

sc.stop()