Ad

Python 3 Multiprocessing - How To Execute A Single Task

Much appreciated in advance !

Task Description: I would like to use Python to collect the free https proxy server information and test. The codes takes a few minutes to run (about 100 proxy servers to test), I understand multiprocessing can remarkably increasing the executing speed, however, I tried a few days but no luck ... seems that all sub-processes are doing the duplicate tasks, rather than doing together.

Need assistance here ...

Codes:

import requests
import re
import telnetlib
import multiprocessing

def run(info1, info2):
    try:
        tn = telnetlib.Telnet(info1, port= info2, timeout= 2)
    except:
        print('not working !')
    else:
        proxy_server = 'http://' + info1 + ':' + info2
        print(proxy_server)

if __name__=='__main__':
    headers = {
        'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/54.0.2840.99 Safari/537.36'}
    html = requests.get('https://www.sslproxies.org', headers=headers, allow_redirects=False)
    pattern = re.compile('<td>(\d.*?)</td><td>(\d+)</td>', re.S)
    items = re.findall(pattern, html.text)
    for item in items:
        for i in range(5):
            p = multiprocessing.Process(target=run(item[0], item[1]), args=('msc%s' % i,))
            p.start()
Ad

Answer

The "for i in range(5)" was spawning the same process 5 times, creating repetition. This can be achieved simply by removing it and creating the process in a simpler way.

    for item in items:
            p = multiprocessing.Process(target=run, args=(item[0], item[1],))
            p.start()
Ad
source: stackoverflow.com
Ad