Ad

Python Crawler Issue

- 1 answer

I have a problem that I can't solve myself as it seems, I hope someone here might have another idea that can help me.

My plan is to crawl the data from comtrade for several countries and timeframes, but even my first call isn't working. The URL I want to send a get request to is http://comtrade.un.org/api/get?&r=32&freq=A&ps=2013&px=H4&cc=AG6&type=C&rg=2&p=0&head=M and if I enter this data in postman I get a proper response with plenty datasets, but if I try to from python I get the response

"{'Message': 'Empty parameters or null values are not permitted. For more information please visit http://comtrade.un.org/data/doc/api/'}"

instead. The API doesn't take any authentication and I didn't set any headers or did any other kind of change to postman, but there it works.

Please take a look at my code and tell me what I'm doing wrong. Did I miss something?

You can try it yourself using the above mentioned URL up to 100 times per hour, maybe you find a way to do so :)

My code:

import json
import requests
url = "http://comtrade.un.org/api/get?&r=32&freq=A&ps=2013&px=H4&cc=AG6&type=C&rg=2&p=0&head=M"
f = requests.get(url, timeout=300)
x = json.loads(f.text)
print(x)
Ad

Answer

The url is malformed, you should replace the ?& with ?, so the correct url becomes: https://comtrade.un.org/api/get?r=32&freq=A&ps=2013&px=H4&cc=AG6&type=C&rg=2&p=0&head=M

Ad
source: stackoverflow.com
Ad