Блог

Я хочу соскрести «Отсутствует:» (помечено красным на изображении)

 strlink= 'https://www.google.com/search?q={0}'.format(link)
            print(strlink)
            url = strlink
            reqs = requests.get(url)
            soup = BeautifulSoup(reqs.text, 'html.parser')
            print(url)
            time.sleep(10)
        try:
            root = lxml.html.parse(url)
            price = root.xpath('//*[@id="rso"]/div[1]/div/div/div[2]/div//span/@text')[0]
            print(price)
        except:
            price="None"
            print(price)
 

«]2]2

https://www.google.com/search?q=AGL CO LOCATION 76-78 PITT ST SYDNEY NSWamp;oq=AGL CO LOCATION 76-78 PITT ST SYDNEY NSWamp;aqs=chrome..69i57.414j0j7amp;sourceid=chromeamp;ie=UTF-8

Чтобы получить правильный ответ от сервера Google, установите User-Agent HTTP-заголовок:

 import requests
from bs4 import BeautifulSoup


url = "https://www.google.com/search"
params = {
    "q": "AGL CO LOCATION 76-78 PITT ST SYDNEY NSW",
    "hl": "en",
}
headers = {
    "User-Agent": "User-Agent: Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:88.0) Gecko/20100101 Firefox/88.0"
}
soup = BeautifulSoup(
    requests.get(url, params=params, headers=headers).content, "html.parser"
)

for result in soup.select(".g"):
    print(result.h3.text)
    print("Missing: ")
    for s in result.select("s"):   # <--- print missing values here
        print(s.text, end=" ")
    print()
    print("-" * 80)
 

С принтами:

 76-78 Pitt Street, Sydney NSW 2000 - Sold Other Property ...
Missing: 
AGL CO 
--------------------------------------------------------------------------------
Sold Other Property at 76-78 Pitt Street, Sydney, NSW 2000 ...
Missing: 
AGL CO 
--------------------------------------------------------------------------------
Telstra readies $300m telephone exchange for sale - AFR
Missing: 
AGL 
--------------------------------------------------------------------------------
Property bigwigs sign up for Telstra's Pitt St exchange auction
Missing: 
AGL NSW 
--------------------------------------------------------------------------------
Pitt Telephone Exchange - The Skyscraper Center
Missing: 
AGL CO NSW 
--------------------------------------------------------------------------------
Telstra to sell Sydney Pitt Street Telephone Exchange with ...
Missing: 
AGL 
--------------------------------------------------------------------------------
Sydney Central - Visit Sydney Australia
Missing: 
CO 76-78 
--------------------------------------------------------------------------------
Michael | Pitt Street, Sydney, NSW | White Pages®
Missing: 
AGL 
--------------------------------------------------------------------------------
Media Briefs - SAGE Journals
Missing: 

--------------------------------------------------------------------------------