Python Csvwriter Outputs Strings On Seperate Rows Instead Of Being On The Same Row
Been a while and i'm getting back into coding for a research project, i'm currently doing a practice site to see what I need to do for the actual site.
I've got everything working how I want, but when the scraped data is outputted to csv it puts the value into a new row instead of the column beside the row it was meant to be on.
I've added links to the output below. Let me know what I need changing as I can't figure it out.
import csv, re
import requests
from bs4 import BeautifulSoup
URL = "https://realpython.github.io/fake-jobs/"
page = requests.get(URL)
with open('testScraperEX.csv', 'w') as f:
write = csv.writer(f)
soup = BeautifulSoup(page.content, "html.parser")
write.writerow(['Title', 'Company', 'Location'])
results = soup.find(id="ResultsContainer")
job_elements = results.find_all("div", class_="card-content")
for job_element in job_elements:
title_element = job_element.find("h2", class_="title")
company_element = job_element.find("h3", class_="company")
location_element = job_element.find("p", class_="location")
Title = title_element.text.strip()
Company = company_element.text.strip()
Location = location_element.text.strip()
write.writerows([[Title],[Company],[Location]])
This is the current output
This is how I want the output to be
Thanks :)
Answer
import csv
import requests
from bs4 import BeautifulSoup
URL = "https://realpython.github.io/fake-jobs/"
page = requests.get(URL)
with open('testScraperEX.csv', 'w', newline='') as f:
write = csv.writer(f)
soup = BeautifulSoup(page.content, "html.parser")
write.writerow(['Title', 'Company', 'Location'])
results = soup.find(id="ResultsContainer")
job_elements = results.find_all("div", class_="card-content")
for job_element in job_elements:
title_element = job_element.find("h2", class_="title")
company_element = job_element.find("h3", class_="company")
location_element = job_element.find("p", class_="location")
Title = title_element.text.strip()
Company = company_element.text.strip()
Location = location_element.text.strip()
write.writerow([Title, Company, Location])
output in the csv file:
Title,Company,Location
Senior Python Developer,"Payne, Roberts and Davis","Stewartbury, AA"
Energy engineer,Vasquez-Davidson,"Christopherville, AA"
Legal executive,"Jackson, Chambers and Levy","Port Ericaburgh, AA"
Fitness centre manager,Savage-Bradley,"East Seanview, AP"
Product manager,Ramirez Inc,"North Jamieview, AP"
... and many more lines ...
This line in your code
write.writerows([[Title],[Company],[Location]])
writes 3 rows, because each element unnecessarily is inside a list (i.e. you have list of 3 one-element lists). This
write.writerow([Title, Company, Location])
writes one line - a list of 3 elements.
Note wrte.writerows
vs write.writerow
and [[Title],[Company],[Location]]
vs [Title, Company, Location]
Related Questions
- → What are the pluses/minuses of different ways to configure GPIOs on the Beaglebone Black?
- → Django, code inside <script> tag doesn't work in a template
- → React - Django webpack config with dynamic 'output'
- → GAE Python app - Does URL matter for SEO?
- → Put a Rendered Django Template in Json along with some other items
- → session disappears when request is sent from fetch
- → Python Shopify API output formatted datetime string in django template
- → Can't turn off Javascript using Selenium
- → WebDriver click() vs JavaScript click()
- → Shopify app: adding a new shipping address via webhook
- → Shopify + Python library: how to create new shipping address
- → shopify python api: how do add new assets to published theme?
- → Access 'HTTP_X_SHOPIFY_SHOP_API_CALL_LIMIT' with Python Shopify Module