Блог

Это ссылка, которую я пытаюсь очистить в качестве примера: Новости Livemint

Вот функция, которая пытается этого достичь:

     t = []
try:
    temp = []
    data = bs.find_all(class_=['contentSec'])
    # logging.info(data)
    for i in data:
        temp = temp   (i.find_all('p'))
    for i in temp:
        t.append(i.get_text())
except Exception as e:
    print(e)
return t
  

Что происходит, если я включаю text = True в параметр find all, он игнорирует пункты со ссылкой (с тегом href).
В противном случае это дает мне огромные пробелы в поле содержимого, вероятно, потому, что реклама на веб-сайте также находится в теге para. Я приложил пример вывода.

Чего я не понимаю?

Данные, которые вы ищете (т. Е. содержимое статьи) доступно непосредственно в исходном коде страницы в div with классе mainArea . Все, что вам нужно сделать, это получить текст этого div и очистить его. Я не думаю, что для получения требуемых данных вообще есть необходимость находить script тег и использовать json module. Но если вам требуются такие данные, как datePublished, ответ @chitown88 будет более полным.

 from bs4 import BeautifulSoup
import requests
url='https://www.livemint.com/Companies/Ot1UTmQ8EMe0DTWSiJCgfJ/Google-teams-with-HDFC-Bank-ICICI-others-for-instant-loans.html'
r=requests.get(url)
soup=BeautifulSoup(r.text,'html.parser')
data=soup.find('div',class_='mainArea')
#let's just clen up the data
cleaned_data=data.text.split('nn')[0].strip()
print(cleaned_data)
  

Вывод

 New Delhi: Google India on Tuesday said it has rebranded its Indian payments app Tez as Google Pay and is partnering four banks to provide instant loans for the app’s users. In the coming weeks, Google Pay users will be able to access customised loans from HDFC Bank Ltd, ICICI Bank Ltd, Federal Bank and Kotak Mahindra Bank Ltd with minimal paperwork, said Caesar Sengupta, vice-president of Google’s Next Billion Users Initiative and Payments, at the Google for India event in New Delhi. Once users holding accounts with these banks accept the bank’s terms, the money will be transferred to their accounts.“We have learnt that when we build for India, we build for the world, and we believe that many of the innovations and features we have pioneered with Tez will work globally," Caesar Sengupta said.Google Tez, which was launched in September, will also expand services for merchants and retailers. About 15,000 retail stores in India will have Google Pay enabled by Diwali 2018, Caesar Sengupta said.Google claims that over 1.2 million small businesses in India are already using Google Pay. In a bid to help their business grow further, Google is building a dedicated merchant experience where they will be discovered through Google Search and Maps, and communicate with their customers through messages and offers.“We are testing these features with merchants in Bangalore and Delhi, and on-boarding more neighbourhoods in the following months," said Sengupta.Google Pay has rivals in Paytm and Facebook Inc.’s WhatsApp targeting the Indian payments market. On Tuesday, Mint reported that Warren Buffett’s Berkshire Hathaway Inc. has sealed a deal with Paytm, marking the legendary investor’s first investment in the country. A string of other big-name players are also expanding in India’s digital payments market including its banks, India Post Payments Bank, and Mukesh Ambani’s Reliance Jio.“The real competition is actually user habits and cash," said Sengupta. “So, all of us (referring to the other players) are all in many ways brothers-in-arms who are trying to move people’s habits away from cash to digital so that we can move India to a digital economy. At Google, we focus on the users so we don’t think so much of the competition."Since its launch, over 55 million people have downloaded Google Tez and more than 22 million people and businesses actively use the app for digital transactions every month, according to company data and figures quoted by Sengupta. Collectively, they have made more than 750 million transactions, with an annual run rate of over $30 billion.The search giant also announced other initiatives including expanding its Google Station internet access programme to 12,000 villages and cities across Andhra Pradesh, potentially reaching 10 million people; the launch of Project Navlekha, where Google will work with Indian publishers to bring more relevant content online; and a new feature in Google Go app that can pull up any webpage and let users listen to it with each word lighting up as it is read.
  

 import requests
import bs4
import pandas as pd
import json

list_of_urls = ['https://www.livemint.com/Companies/Ot1UTmQ8EMe0DTWSiJCgfJ/Google-teams-with-HDFC-Bank-ICICI-others-for-instant-loans.html']

headers = {'user-agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/72.0.3626.119 Safari/537.36'}

results_df = pd.DataFrame()


for url in list_of_urls:
    response = response = requests.get(url, headers=headers)
    soup = bs4.BeautifulSoup(response.text, 'html.parser')

    scripts = soup.find_all('script',{'type':'application/ld json'})
    for script in scripts:
        if '"headline"' in script.text:
            jsonStr = script.text.strip()
            jsonObj = json.loads(jsonStr)

            date_pub = jsonObj['datePublished']
            date_mod = jsonObj['dateModified']
            data = jsonObj['articleBody']
            url = jsonObj['url']

            temp_df = pd.DataFrame([[date_pub, data, url]], columns=['date_published','data','url'])

            results_df = results_df.append(temp_df)

results_df = results_df.reset_index(drop=True)
results_df.to_csv('path/to/file.csv', index=False)