#1
  1. No Profile Picture
    Registered User
    Devshed Newbie (0 - 499 posts)

    Join Date
    Mar 2013
    Posts
    2
    Rep Power
    0

    Dnspython - Increasing occurrence of timeout exception


    I managed to write this script which tests list of domains. His task is to get answer from every domain if it has any NS record, if it has, then this script tests every obtained NS records if it has IPv6 address (AAAA record).
    Problem is that script more and more triggers Timeout exception although the domain has NS records. In the last stages is nearly every domain evaluated as without NS records.
    I think problem is with threads or in main_thread part of the script.
    I used threading because I need to process 280k domains and I need to process it fast. So, I created for every domain thread (main_thread) and within the main thread Subthreads (NS_getAAAA_thread) for every obtained NS record.

    Main code:

    Code:
    resolver = dns.resolver.Resolver()
    resolver.lifetime= 2.5
    resolver.timeout= 2
    
    lock = threading.Lock()
    threads = []
    
    queue = Queue.Queue()
    MAX_THREAD_COUNT = 100
    
    
    try:
        cursor.execute("SELECT * FROM domains")
        results = cursor.fetchall()
    
        for i in range(MAX_THREAD_COUNT):
            t = main_thread(queue)
            t.daemon = True
            threads.append(t)
            t.start()
    
        for row in results:
            queue.put((row[0],row[1]))
    
        queue.join()
    
        for thread in threads:
            thread.keepRunning = False
        threads = []
    
    except Exception as ex:
        sys.stderr.write("Global Error: " + str(ex))
    Main thread code:

    Code:
    class main_thread(threading.Thread):
        def __init__(self,queue):
            threading.Thread.__init__(self)
            self.queue = queue
    
        def run(self):
            self.keepRunning = True
            while self.keepRunning:
                arg = self.queue.get()
                id_domain = arg[0]
                domain = arg[1]
                SubThreads = []
    
                #---------------- NS Records ----------------
                try:
                    answers = resolver.query(domain, 'NS') 
                    for ns in answers:
                        t = NS_getAAAA_thread(id_domain, ns)
                        t.setDaemon(True)
                        SubThreads.append(t)
                        t.start()
    
                except NXDOMAIN as enx:
                    print "NS - Error: NXDOMAIN - domain does not exist"
                except Timeout as etime:
                    print "NS - Error: dns.exception.Timeout"
                except NoAnswer as ea:
                    print "NS - Error: No Answer"
                except NoNameservers as ens:
                    print "NS - Error: No non-broken nameservers are available"
    
    
                for SubThread in SubThreads:
                    SubThread.join()
    
                self.queue.task_done()
    NS thread code:

    Code:
    class NS_getAAAA_thread(threading.Thread):
        def __init__(self,id_domain,ns):
            threading.Thread.__init__(self)
            self.id_domain = id_domain
            self.ns = ns
    
        def run(self):
            res = dns.resolver.Resolver()
            res.lifetime = 2
            res.timeout = 1
    
            try:
                answers = res.query(str(self.ns),'AAAA')
                ip_ns = str(answers[0])
            except DNSException:
                ip_ns = "N/A"
            except Exception:
                sys.stderr.write("Global Error: " + str(ex))
    
            with lock:
                sql = "INSERT INTO ns (id_domain,ns,ip_ns) VALUES ('" + str(self.id_domain) + "','" + str(self.ns) + "','" + str(ip_ns) + "')"
                try:
                    cursor.execute(sql)
                    db.commit()
                    print "NS" , '>>' , str(ip_ns), ' :: ', str(self.ns)
                except Exception as ex:
                    sys.stderr.write("Global Error: " + str(ex))
                    db.rollback()
  2. #2
  3. No Profile Picture
    Contributing User
    Devshed Newbie (0 - 499 posts)

    Join Date
    Feb 2013
    Posts
    138
    Rep Power
    2
    I haven't looked at the code yet, but I thought of another thing. Have you checked the dns server to see if it likes to process 280k requests from the same ip?
  4. #3
  5. No Profile Picture
    Registered User
    Devshed Newbie (0 - 499 posts)

    Join Date
    Mar 2013
    Posts
    2
    Rep Power
    0
    Originally Posted by partoj
    I haven't looked at the code yet, but I thought of another thing. Have you checked the dns server to see if it likes to process 280k requests from the same ip?
    it could be the problem but when I am requesting just AAAA records for www.[domain] it doesnt do the same thing, if there was an exception it was triggered correctly. If I test those domains it truly doesnt have AAAA record unlike the NS records.

    btw. How can I check the dns server if it doesnt have any limit or restrictions?

IMN logo majestic logo threadwatch logo seochat tools logo