Ad

Python Csvwriter Outputs Strings On Seperate Rows Instead Of Being On The Same Row

- 1 answer

Been a while and i'm getting back into coding for a research project, i'm currently doing a practice site to see what I need to do for the actual site.

I've got everything working how I want, but when the scraped data is outputted to csv it puts the value into a new row instead of the column beside the row it was meant to be on.

I've added links to the output below. Let me know what I need changing as I can't figure it out.

import csv, re
import requests 
from bs4 import BeautifulSoup

URL = "https://realpython.github.io/fake-jobs/"
page = requests.get(URL)
with open('testScraperEX.csv', 'w') as f:
    write = csv.writer(f)
    soup = BeautifulSoup(page.content, "html.parser")
    write.writerow(['Title', 'Company', 'Location'])
    results = soup.find(id="ResultsContainer")
    job_elements = results.find_all("div", class_="card-content")
    for job_element in job_elements:
        title_element = job_element.find("h2", class_="title")
        company_element = job_element.find("h3", class_="company")
        location_element = job_element.find("p", class_="location")
        Title = title_element.text.strip()
        Company = company_element.text.strip()
        Location = location_element.text.strip()
        write.writerows([[Title],[Company],[Location]])

This is the current output 1

This is how I want the output to be 2

Thanks :)

Ad

Answer

import csv
import requests 
from bs4 import BeautifulSoup

URL = "https://realpython.github.io/fake-jobs/"
page = requests.get(URL)
with open('testScraperEX.csv', 'w', newline='') as f:
    write = csv.writer(f)
    soup = BeautifulSoup(page.content, "html.parser")
    write.writerow(['Title', 'Company', 'Location'])
    results = soup.find(id="ResultsContainer")
    job_elements = results.find_all("div", class_="card-content")
    for job_element in job_elements:
        title_element = job_element.find("h2", class_="title")
        company_element = job_element.find("h3", class_="company")
        location_element = job_element.find("p", class_="location")
        Title = title_element.text.strip()
        Company = company_element.text.strip()
        Location = location_element.text.strip()
        write.writerow([Title, Company, Location])

output in the csv file:

Title,Company,Location
Senior Python Developer,"Payne, Roberts and Davis","Stewartbury, AA"
Energy engineer,Vasquez-Davidson,"Christopherville, AA"
Legal executive,"Jackson, Chambers and Levy","Port Ericaburgh, AA"
Fitness centre manager,Savage-Bradley,"East Seanview, AP"
Product manager,Ramirez Inc,"North Jamieview, AP"
... and many more lines ...

This line in your code

write.writerows([[Title],[Company],[Location]])

writes 3 rows, because each element unnecessarily is inside a list (i.e. you have list of 3 one-element lists). This

write.writerow([Title, Company, Location]) 

writes one line - a list of 3 elements.

Note wrte.writerows vs write.writerow and [[Title],[Company],[Location]] vs [Title, Company, Location]

Ad
source: stackoverflow.com
Ad