Блог

Я изучал веб-скрапинг и выбираю наборы данных YouTube, поскольку они легко доступны

Как я могу записать свои очищенные выходные данные в CSV, поскольку я так запутался в том, где мне следует использовать write.writerows?

Пример кода ниже:

 from selenium import webdriver
from bs4 import BeautifulSoup
import csv

urls = [
    'https://www.youtube.com/channel/UC8butISFwT-Wl7EV0hUK0BQ',
    'https://www.youtube.com/user/thenewboston',
    'https://www.youtube.com/user/gotreehouse',
    'https://www.youtube.com/user/derekbanas',
    'https://www.youtube.com/channel/UCWr0mx597DnSGLFk1WfvSkQ',
    'https://www.youtube.com/user/ProgrammingKnowledge'
  ]

def scriptscrapejprh():
    driver = webdriver.Chrome('C:/Users/Phong/Desktop/chromedriver.exe')
    for url in urls:
        driver.get('{}/videos?view=0amp;sort=pamp;flow=grid'.format(url))
        content = driver.page_source.encode('utf-8').strip()
        soup = BeautifulSoup(content, 'lxml')
        titles = soup.findAll('a',id='video-title')
        views = soup.findAll('span',class_='style-scope ytd-grid-video-renderer')
        video_urls = soup.findAll('a',id='video-title')
        print('Channel: {}'.format(url))
        i = 0 # views and time
        j = 0 # urls
        for title in titles[:20]:
            print('n{}t{}t{}thttps://www.youtube.com{}'.format(title.text, views[i].text, 
views[i 1].text, video_urls[j].get('href')))
            i =2
            j =1               

scriptscrapejprh()
  

И это пример вывода (я не могу включить ссылки):

 Channel: ProgrammingKnowledge

C Programming Tutorial | Learn C programming | C language       5.5M views      4 years ago     -link-

Java Tutorial For Beginners 1 - Introduction and Installing the java (JDK) Step by Step Tutorial    4.1M views  5 years ago     -link-

How to create Partition on Windows 10 | Partition Hard Drives   3.2M views      3 years ago     -link-

Android Tutorial for Beginners 1 # Introduction and Installing and Configuring Java JDK 2.9M views  5 years ago -link-

How to Install Java JDK on Windows 10 ( with JAVA_HOME )        2M views        5 years ago     -link-

How to Convert Video Files using VLC Media Player       1.6M views      6 years ago     -link-

How To Install MySQL on Windows 10      1.6M views      3 years ago     -link-

Python Tutorial for Beginners 1 - Getting Started and Installing Python (For Absolute Beginners)    1.6M views  6 years ago     -link-

Beginners MS Access Database Tutorial 1 - Introduction and Creating Database    1.6M views      6 years ago     
  

Вы можете использовать writerows функцию внутри второго цикла. Попробуйте следующий код:

 from selenium import webdriver
from bs4 import BeautifulSoup
import csv

urls = [
    'https://www.youtube.com/channel/UC8butISFwT-Wl7EV0hUK0BQ',
    'https://www.youtube.com/user/thenewboston',
    'https://www.youtube.com/user/gotreehouse',
    'https://www.youtube.com/user/derekbanas',
    'https://www.youtube.com/channel/UCWr0mx597DnSGLFk1WfvSkQ',
    'https://www.youtube.com/user/ProgrammingKnowledge'
  ]

data_file = open("data.csv", "w", newline="", encoding='utf-8')
writer = csv.writer(data_file, delimiter=",")


def scriptscrapejprh():
    driver = webdriver.Chrome('C:/Users/Phong/Desktop/chromedriver.exe')
    for url in urls:
        driver.get('{}/videos?view=0amp;sort=pamp;flow=grid'.format(url))
        content = driver.page_source.encode('utf-8').strip()
        soup = BeautifulSoup(content, 'lxml')
        titles = soup.findAll('a',id='video-title')
        views = soup.findAll('span',class_='style-scope ytd-grid-video-renderer')
        video_urls = soup.findAll('a',id='video-title')
        print('Channel: {}'.format(url))
        i = 0 # views and time
        j = 0 # urls
        for title in titles[:20]:
            print('n{}t{}t{}thttps://www.youtube.com{}'.format(title.text, views[i].text, 
views[i 1].text, video_urls[j].get('href')))
            writer.writerow([title.text, views[i].text, views[i 1].text, "https://www.youtube.com/{0}".format(video_urls[j].get('href'))])
            i =2
            j =1

    data_file.close()

scriptscrapejprh()