Блог

Я применяю определяемую пользователем функцию к фрейму данных spark, как показано ниже,

 @udf("double")
def discount_udf (row):
  if ((row['total_order'] == 2) or (row['total_order'] == 3)):
    return 2.50
  elif ((row['total_order'] == 4) or (row['total_order'] == 5)):
    return 1.20
  elif ((row['total_order'] == 6) or (row['total_order'] == 7)):
    return 0.60
  elif ((row['total_order'] == 8) or (row['total_order'] == 9) or (row['total_order'] == 10) or (row['total_order'] == 11)):
    return 0.00
  elif ((row['total_order'] == 12) or (row['total_order'] == 13) or (row['total_order'] == 14) or (row['total_order'] == 15)):
    return -0.20
  elif ((row['total_order'] == 16) or (row['total_order'] == 17) or (row['total_order'] == 18) or (row['total_order'] == 19) or (row['total_order'] == 20) or (row['total_order'] == 21) or (row['total_order'] == 22) or (row['total_order'] == 23)):
    return -0.20
  elif ((row['total_order'] == 24) or (row['total_order'] == 25) or (row['total_order'] == 26) or (row['total_order'] == 27) or (row['total_order'] == 28) or (row['total_order'] == 29) or (row['total_order'] == 30) or (row['total_order'] == 31)):
    return -0.40
  else :
    return -0.50

from pyspark.sql.functions import udf
df.withColumn("discount_rate", discount_udf(F.col('total_order')))
 

Однако это приводит меня к этой ошибке

Ошибка

 PythonException: 'TypeError: 'float' object is not subscriptable', from <command-1374686736879751>, line 3. Full traceback below:
org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 100.0 failed 4 times, most recent failure: Lost task 0.3 in stage 100.0 (TID 164) (10.139.64.4 executor 0): org.apache.spark.api.python.PythonException: 'TypeError: 'float' object is not subscriptable', from <command-1374686736879751>, line 3. Full traceback below:
Traceback (most recent call last):
  File "<command-1374686736879751>", line 3, in discount_udf
TypeError: 'float' object is not subscriptable
 

Я попробовал ту же функцию с фреймом данных «панды», она отлично работает для меня.

 df['discount_rate_1'] = df.apply(discount_udf, axis=1)
 

Кто-нибудь может помочь/подсказать, что здесь не так?

Заранее спасибо

Я изменяю начало и позволяю тебе сделать все остальное :

 @udf("double")
def discount_udf (total_order):
  if ((total_order == 2) or (total_order == 3)):
 

Вам просто нужно заменить каждый row['total_order'] из total_order них .

Я бы посоветовал вам также внести это изменение :

 if ((total_order == 2) or (total_order == 3)):
# TO BECOME
if total_order  in (2,3):
# OR ALSO POSSIBLE
if 2 <= total_order <= 3: # It is not exactly the same but should work if you only have integer