Блог

Я не могу получить текущее поле «тип«, используя BeautifulSoup . Текущий код выводит пустое значение для переменной «type»:

Код:

 from bs4 import BeautifulSoup
import requests

url='https://ash.confex.com/ash/2021/webprogram/Session20851.html'

res = requests.get(url)
soup = BeautifulSoup(res.content, 'html.parser')

content = soup.find_all('div', class_='paper')

for property in content:
    title = property.find('div',class_='cricon').text
    type_ = property.find("div",{"id":"info"})
 

Как вы можете видеть здесь, это выглядит как переменная «свойства» во время каждой итерации содержимого.

 <div class="paper">
<div class="papertime">9:30 AM</div>
<div class="papernumber"><a href="Paper146905.html">7</a></div>
<div class="papertitle">
<div class="cricon"><a href="Paper146905.html">Sustained Improvements in Patient-Reported Quality of Life up to 24 Months Post-Treatment with LentiGlobin for Sickle Cell Disease (bb1111) Gene Therapy</a></div>
<span class="paperauthors">
<p class="name"><b>Mark C. Walters, MD</b><sup>1</sup>, John F. Tisdale, MD<sup>2</sup><sup>*</sup>, Markus Y. Mapara, MD, PhD<sup>3</sup>, Lakshmanan Krishnamurti, MD<sup>4</sup>, Janet L. Kwiatkowski, MD, MSCE<sup>5,6</sup>, Banu Aygun, MD<sup>7</sup>, Kimberly A. Kasow, DO<sup>8</sup><sup>*</sup>, Stacey Rifkin-Zenenberg, DO<sup>9</sup>, Jennifer Jaroscak, MD<sup>10</sup>, Diana Garbinsky, MS<sup>11</sup><sup>*</sup>, Costel Chirila, PhD<sup>11</sup><sup>*</sup>, Meghan E. Gallagher, MSc<sup>12</sup><sup>*</sup>, Xinyan Zhang, PhD<sup>12</sup><sup>*</sup>, Pei-Ran Ho, MD<sup>12</sup><sup>*</sup>, Alexis A. Thompson, MD, MPH<sup>13,14</sup> and Julie Kanter, MD<sup>15</sup></p><p class="address"><sup>1</sup>Division of Hematology, UCSF Benioff Children's Hospital Oakland, Oakland, CA<br/><sup>2</sup>Cellular and Molecular Therapeutics Branch NHLBI/NIDDK, National Institutes of Health, Bethesda, MD<br/><sup>3</sup>Division of Hematology/Oncology, Columbia Center for Translational Immunology, Columbia University Medical Center, New York, NY<br/><sup>4</sup>Aflac Cancer and Blood Disorders Center, Department of Pediatrics, Emory Healthcare, Atlanta, GA<br/><sup>5</sup>Division of Hematology, Children's Hospital of Philadephia, Philadelphia, PA<br/><sup>6</sup>Department of Pediatrics, University of Pennsylvania Perelman School of Medicine, Philadelphia, PA<br/><sup>7</sup>Cohen Children’s Medical Center, Queens, NY<br/><sup>8</sup>University of North Carolina, Chapel Hill<br/><sup>9</sup>Hackensack University Medical Center, Hackensack, NJ<br/><sup>10</sup>University Medical Center, Medical University of South Carolina Health, Charleston, SC<br/><sup>11</sup>RTI Health Solutions, Research Triangle Park, NC<br/><sup>12</sup>bluebird bio, Inc., Cambridge, MA<br/><sup>13</sup>Feinberg School of Medicine, Northwestern University, Chicago, IL<br/><sup>14</sup>Ann amp;amp; Robert H. Lurie Children’s Hospital of Chicago, Chicago, IL<br/><sup>15</sup>University of Alabama Birmingham, Birmingham, AL</p>
</span>
<div class="location"></div>
<div class="media">
</div>
</div>
</div>
 

Другими словами, вы повторяете каждое событие, но вам нужно получить только заголовок div , называемый «info» в ID.

Это должно сработать:

 from bs4 import BeautifulSoup
import requests

url = 'https://ash.confex.com/ash/2021/webprogram/Session20851.html'

res = requests.get(url)
soup = BeautifulSoup(res.content,'html.parser')

content = soup.find_all('div', class_='paper')
info = soup.find_all('div', class_ ='datetime')
type_ = soup.find("span", string="Type:").next_sibling

for property in content:
    title = property.find('div',class_='cricon').text
    print(title, type_, sep="n", end="nn")
 

Вывод:

 Sustained Improvements in Patient-Reported Quality of Life up to 24 Months Post-Treatment with LentiGlobin for Sickle Cell Disease (bb1111) Gene Therapy
 Oral

Activation of Pyruvate Kinase-R with Etavopivat (FT-4202) Is Well Tolerated, Improves Anemia, and Decreases Intravascular Hemolysis in Patients with Sickle Cell Disease Treated for up to 12 Weeks
 Oral

Etavopivat, an Allosteric Activator of Pyruvate Kinase-R, Improves Sickle RBC Functional Health and Survival and Reduces Systemic Markers of Inflammation and Hypercoagulability in Patients with Sickle Cell Disease: An Analysis of Exploratory Studies in a Phase 1 Study
 Oral

Mitapivat (AG-348) Demonstrates Safety, Tolerability, and Improvements in Anemia, Hemolysis, Oxygen Affinity, and Hemoglobin S Polymerization Kinetics in Adults with Sickle Cell Disease: A Phase 1 Dose Escalation Study
 Oral

Hydroxyurea Reduces the Transfusion Burden in Children with Sickle Cell Anemia: The Reach Experience
 Oral

Initial Safety and Efficacy Results from the Phase II, Multicenter, Open-Label Solace-Kids Trial of Crizanlizumab in Adolescents with Sickle Cell Disease (SCD)
 Oral
 

Если вы ищете только «устный» текст, то вы ищете это:

 result = soup.select_one('#info span:nth-child(3)').next_sibling.strip()

# Oral
 

Если вы хотите выполнить итерацию по всем результатам:

 for result in soup.select('.cricon a'):
    title = result.text
    type_ = soup.select_one('#info span:nth-child(3)').next_sibling.strip()

    print(title, type_, sep='n')

'''
Sustained Improvements in Patient-Reported Quality of Life up to 24 Months Post-Treatment with LentiGlobin for Sickle Cell Disease (bb1111) Gene Therapy
Oral
Activation of Pyruvate Kinase-R with Etavopivat (FT-4202) Is Well Tolerated, Improves Anemia, and Decreases Intravascular Hemolysis in Patients with Sickle Cell Disease Treated for up to 12 Weeks
Oral

# other results..
'''
 

Полный код и пример в онлайн-среде разработки:

 from bs4 import BeautifulSoup
import requests

url='https://ash.confex.com/ash/2021/webprogram/Session20851.html'

res = requests.get(url)
soup = BeautifulSoup(res.content,'html.parser')

result = soup.select_one('#info span:nth-child(3)').next_sibling.strip()
print(result)
# Oral

# ----------------

for result in soup.select('.cricon a'):
    title = result.text
    type_ = soup.select_one('#info span:nth-child(3)').next_sibling.strip()

    print(title, type_, sep='n')


'''
Sustained Improvements in Patient-Reported Quality of Life up to 24 Months Post-Treatment with LentiGlobin for Sickle Cell Disease (bb1111) Gene Therapy
Oral
Activation of Pyruvate Kinase-R with Etavopivat (FT-4202) Is Well Tolerated, Improves Anemia, and Decreases Intravascular Hemolysis in Patients with Sickle Cell Disease Treated for up to 12 Weeks
Oral
...
'''
 

P.S. У меня есть специальный блог по очистке веб-страниц. Если вам нужно проанализировать поисковые системы, попробуйте использовать SerpApi.

Отказ от ответственности, я работаю в SerpApi.