Блог

У меня проблема в scala. Мне нужно удалить стоп-слова из текстового файла RDD [String].

 val sc = new SparkContext(conf)

val tweetsPath = args(0)
val outputDataset = args(1)

val tweetsRaw: RDD[String] = sc.textFile(tweetsPath)

val stopWords = Array("a","able","about","across","after","all","almost","also","am","among","an","and","any","are","as","at","be","because","been","but","by","can","cannot","could","dear","did","do","does","either","else","ever","every","for","from","get","got","had","has","have","he","her","hers","him","his","how","however","i","if","in","into","is","it","its","just","least","let","like","likely","may","me","might","most","must","my","neither","no","nor","not","of","off","often","on","only","or","other","our","own","rather","said","say","says","she","should","since","so","some","than","that","the","their","them","then","there","these","they","this","tis","to","too","twas","us","wants","was","we","were","what","when","where","which","while","who","whom","why","will","with","would","yet","you","your","ain't","aren't","can't","could've","couldn't","didn't","doesn't","don't","hasn't","he'd","he'll","he's","how'd","how'll","how's","i'd","i'll","i'm","i've","isn't","it's","might've","mightn't","must've","mustn't","shan't","she'd","she'll","she's","should've","shouldn't","that'll","that's","there's","they'd","they'll","they're","they've","wasn't","we'd","we'll","we're","weren't","what'd","what's","when'd","when'll","when's","where'd","where'll","where's","who'd","who'll","who's","why'd","why'll","why's","won't","would've","wouldn't","you'd","you'll","you're","you've")

val cleanTxt = tweetsRaw.
  filter(x => x.startsWith("San Francisco") || x.startsWith("Chicago") || !stopWords.contains(x));

cleanTxt.saveAsTextFile(outputDataset)
 

Я пробовал это, но это не работает. Я должен поддерживать ту же структуру (используя SparkConf и не переходя на SparkSession). Как я могу выбрать все твиты, начинающиеся с «Чикаго» и «Сан-Франциско», удалив стоп-слова из текста и получив на выходе все твиты построчно без этих стоп-слов?

Я сделал плоскую карту моего tweetsraw, но с помощью плоской карты я получаю на выходе только слова без стоп-слов, но мне нужна целая строка без стоп-слов, а не только слова.

Я надеюсь, что мне было ясно, чего я хотел, и что вы могли бы помочь мне решить эту проблему!

Спасибо, ребята.

PS Я много чего перепробовал с помощью метода StopWordsRemover из библиотеки scala, но я просто не могу понять, как заставить его работать без инициализации SparkSession и выполнения этого с помощью SparkConf.

Как я могу выбрать все твиты, начинающиеся с «Чикаго» и «Сан-Франциско», удалив стоп-слова из текста и получив на выходе все твиты построчно без этих стоп-слов?

Следующая строка в вашем скрипте spark отфильтровывает твиты для ваших условий. Однако это не удаление стоп-слова из строки.

 val cleanTxt = tweetsRaw.
  filter(x => x.startsWith("San Francisco") || x.startsWith("Chicago") || !stopWords.contains(x));
 

Если вы хотите удалить стоп-слова, вам нужно использовать преобразование карты, которое удалит стоп-слова из строки, а затем вы сможете сохранить его в файл.

Предполагая, что каждая строка представляет собой твит, разделенный пробелом, вот что я бы сделал, чтобы удалить стоп-слова.

 cleanTxt.map(tweet => tweet.split(" ").filterNot(x => stop.contains(x)).mkString(" ").saveAsTextFile(outputDataset)