Блог

Я пытаюсь удалить повторяющиеся строки из заданного ввода. И ввод — это полный беспорядок. Например, ниже приведен пример ввода.

 HISTORY
HISTORY TV"
HISTORY TV" THIS
HISTORY TV" THIS WEEKEND
HISTORY TV" THIS WEEKEND ON
HISTORY TV" THIS WEEKEND ON C-SPAN3.
WEEKEND ON C-SPAN3.
WEEKEND ON C-SPAN3. "THE
WEEKEND ON C-SPAN3. "THE MARCH
WEEKEND ON C-SPAN3. "THE MARCH ON
WEEKEND ON C-SPAN3. "THE MARCH ON WASHINGTON"
"THE MARCH ON WASHINGTON" FOR
"THE MARCH ON WASHINGTON" FOR JOBS
"THE MARCH ON WASHINGTON" FOR JOBS AND
"THE MARCH ON WASHINGTON" FOR JOBS AND FREEDOM
"THE MARCH ON WASHINGTON" FOR JOBS AND FREEDOM WAS
"THE MARCH ON WASHINGTON" FOR JOBS AND FREEDOM WAS 49
FOR JOBS AND FREEDOM WAS 49 YEARS
FOR JOBS AND FREEDOM WAS 49 YEARS AGO.
YEARS AGO. ON
YEARS AGO. ON AUGUST
YEARS AGO. ON AUGUST 28th,
YEARS AGO. ON AUGUST 28th, 1963.
ON AUGUST 28th, 1963. THE
ON AUGUST 28th, 1963. THE MARCH
ON AUGUST 28th, 1963. THE MARCH WAS
ON AUGUST 28th, 1963. THE MARCH WAS ORKGANIZED
ON AUGUST 28th, 1963. THE MARCH WAS ORKGANIZED TO
ON AUGUST 28th, 1963. THE MARCH WAS ORKGANIZED TO PUSH
THE MARCH WAS ORKGANIZED TO PUSH FOR
THE MARCH WAS ORKGANIZED TO PUSH FOR COMPREHENSIVE
THE MARCH WAS ORKGANIZED TO PUSH FOR COMPREHENSIVE CIVIL
THE MARCH WAS ORKGANIZED TO PUSH FOR COMPREHENSIVE CIVIL RIGHTS
FOR COMPREHENSIVE CIVIL RIGHTS INCLUDING
FOR COMPREHENSIVE CIVIL RIGHTS INCLUDING PUBLIC
FOR COMPREHENSIVE CIVIL RIGHTS INCLUDING PUBLIC SCHOOL
INCLUDING PUBLIC SCHOOL DESEGREGATION,
INCLUDING PUBLIC SCHOOL DESEGREGATION, VOTING
INCLUDING PUBLIC SCHOOL DESEGREGATION, VOTING RIGHTS
DESEGREGATION, VOTING RIGHTS PROTECTION
  

Вот желаемый результат

 HISTORY TV" THIS WEEKEND ON C-SPAN3.
"THE MARCH ON WASHINGTON" FOR JOBS AND FREEDOM WAS 49
ON AUGUST 28th, 1963. THE MARCH WAS ORKGANIZED TO PUSH
FOR COMPREHENSIVE CIVIL RIGHTS INCLUDING PUBLIC SCHOOL
DESEGREGATION, VOTING RIGHTS PROTECTION
  

который читается человеком и содержит только неповторяющиеся предложения.

ниже приведен код, который у меня есть прямо сейчас

 import os
import re

directory = './reprocess'
for filename in os.listdir(directory):
    
    print("Processing... {}".format(filename))

    with open("./reprocess/" filename, "r") as inputFile:
        data = inputFile.read().splitlines()

    new_data = ""
    
    for line in data:
        new_data = new_data   line   "n"

    cleaned = re.sub(r"(b[w'] (?:[s.] [w'] )*)(?=[s.] 1)[s.] ", 'n', new_data)
    cleaned = re.sub(r"An |(?<=[^.] )n |n (?=n)|n Z", '', cleaned)
    
    with open("./reprocess_out/" filename, "w") as outFile:
        outFile.write(cleaned)
  

и имеют ужасный результат

 HISTORY TV"
HISTORY TV" THIS
HISTORY TV" THIS WEEKEND
HISTORY TV" THIS WEEKEND ON
HISTORY TV" THIS WEEKEND ON C-SPAN3.
WEEKEND ON C-SPAN3.
WEEKEND ON C-SPAN3. "THE
WEEKEND ON C-SPAN3. "THE MARCH
WEEKEND ON C-SPAN3. "THE MARCH ON
WEEKEND ON C-SPAN3. "THE MARCH ON WASHINGTON"
"THE MARCH ON WASHINGTON" FOR
"THE MARCH ON WASHINGTON" FOR JOBS
"THE MARCH ON WASHINGTON" FOR JOBS AND
"THE MARCH ON WASHINGTON" FOR JOBS AND FREEDOM
"THE MARCH ON WASHINGTON" FOR JOBS AND FREEDOM WAS
"THE MARCH ON WASHINGTON" FOR JOBS AND FREEDOM WAS 49 YEARS AGO. ON AUGUST 28th,
YEARS AGO. ON AUGUST 28th, 1963.
ON AUGUST 28th, 1963. THE
ON AUGUST 28th, 1963. THE MARCH
ON AUGUST 28th, 1963. THE MARCH WAS
  

Пожалуйста, дайте мне знать, как я могу решить эту проблему. Это, конечно, сложный вопрос, и это регулярное выражение на самом деле не выполняет ту работу, которую я хотел, чтобы оно выполняло.

Редактировать (не во всех строках есть новые слова):

В этом случае я бы сделал следующее:

 last_words = [data[0]]
for line in data:
    last_word = line.split()[-1] # could be changed to a regex
    if last_words[-1] != last_word:
        last_words.append(last_word)

cleaned = " ".join(last_words)
  

В данном случае data список строк.

Предыдущий ответ:

Согласно вашему комментарию: «каждая строка добавляет только одно слово после последнего».

В этом случае вы можете просто получить последние слова из всех строк:

 res = re.findall("(S*)n", data)
print(" ".join(res))
  

где data ввод в виде одной строки, а не разделен на строки.

Это даст вам желаемый результат только в одной строке.