In the code below, gen_window
has a print statement in it but when I run the code the print statement doesn't get executed. Why is that and then how should I go about debugging such lambda functions? (Even the debugger ignores the breakpoint in these functions.)
getpairs = rdd.flatMap(lambda xi: gen_window(xi, n))
def gen_window(xi, n):
x, i = xi
l = []
for offset in range(n):
print("-->", (i - offset, (i, x)))
l.append((i - offset, (i, x)))
return l
Works:
def gen_window(xi, n):
x, i = xi
l = []
for offset in range(n):
print("-->", (i - offset, (i, x)))
l.append((i - offset, (i, x)))
return l
xi = [3,5]
n = 3
gen_window(xi, n)
Lambdas only get executed wenn you actually use them - If you get no output you are probably never using it.
Output:
--> (5, (5, 3))
--> (4, (5, 3))
--> (3, (5, 3))