Multiprocessing
Page content
Parallel Processing
i recently read an article about parallel processing. i remembered my domain checker service which checks a lot of domains for their availablitly, and this script runs sequentiel and needs around 30 seconds.
initially i worked on a caching mechanism to speed up results. but if a service is not used that often (nobody is useing my domain checker…), there is not much you can gain with caching.
so, i gave a try with Multiprocessing and i have to admit i’m prettyhappy with the result !
https://docs.python.org/3/library/multiprocessing.html
SampleCode Snippet
from multiprocessing import Pool
from pprint import pprint
import dns.resolver
timeout = 1
result = {}
def check_soa_record(domain):
try:
result = dns.resolver.resolve(domain, "SOA", lifetime=timeout)
for rdata in result:
return domain, True
except Exception as e:
return domain, False
def check_a_record(domain):
try:
result = dns.resolver.resolve(domain, "A", lifetime=timeout)
for rdata in result:
return domain, True
except Exception as e:
return domain, False
def check_aaaa_record(domain):
try:
result = dns.resolver.resolve(domain, "AAAA", lifetime=timeout)
for rdata in result:
return domain, True
except Exception as e:
return domain, False
if __name__ == "__main__":
domains = [
"google.com",
"facebook.com",
"twitter.com",
"stackoverflow.com",
]
# Create a Pool with 4 processes
with Pool(4) as p:
# Use the map() method to apply the process_domain function to each element in the domains list
result["soa"] = p.map(check_soa_record, domains)
result["a"] = p.map(check_a_record, domains)
result["aaaa"] = p.map(check_aaaa_record, domains)
pprint(result)
run it
time python test.py
{'a': [('google.com', True),
('facebook.com', True),
('twitter.com', True),
('stackoverflow.com', True)],
'aaaa': [('google.com', True),
('facebook.com', True),
('twitter.com', False),
('stackoverflow.com', False)],
'soa': [('google.com', True),
('facebook.com', True),
('twitter.com', True),
('stackoverflow.com', True)]}
real 0m0.335s
user 0m0.201s
sys 0m0.074s
as you can see, 0.3 Seconds th check 4 Domains for existing A, AAAA & SOA Records … cool :)
Any Comments ?
sha256: 95bc325aa794a1e902a9ec3f98987a332595438ffa4a8092fd46fe54f1736ab8