Ad

Web Crawling: Python Saving File With -o File.json As Utf-8: The Output Shows Characters Like \u00a9

- 1 answer

Using scrapy crawler I am trying to extract data from html page and save the output as json file using command line:

scrapy crawl Product -o test.json

in my code I am using

for desc in response.css('div.pdp__description').extract():
            yield {
                'desc' : desc
            }

but it saves the data using \u00a0 and similar characters, what should I do?

Ad

Answer

Set FEED_EXPORT_ENCODING = 'utf-8' in your settings.py.

Refer to the documentation: https://docs.scrapy.org/en/1.6/topics/feed-exports.html?highlight=FEED_EXPORT_ENCODING#feed-export-encoding

Ad
source: stackoverflow.com
Ad