Develop

README.md

@@ -1,7 +1,9 @@
-pythonwhois
-===========
+The WHOIS Oracle, forked from pythonwhois
+=========================================
 
-A WHOIS retrieval and parsing library for Python.
+Because it is all knowing! 
+A WHOIS retrieval and parsing library for Python, forked from pythonwhois
+and updated by me.
 
 ## Dependencies
 
@@ -11,6 +13,32 @@ None! All you need is the Python standard library.
 
 The manual (including install instructions) can be found in the doc/ directory. A HTML version is also viewable [here](http://cryto.net/pythonwhois).
 
+## Cache configuration
+Using pythonwhois.set_persistent_cache a cache can be set. If a cache is set,
+whois-oracle will look there for WHOIS servers for TLD's. For domains with thin
+WHOIS servers, only the 'head' WHOIS server is cached, not the referral servers.
+Otherwise it would
+be impossible to get the correct information because the information for the domain
+might not be on that WHOIS server at all.
+
+## Cool down configuration
+This feature is not useful for single lookups, but for bulk this comes in really handy.
+Every WHOIS server gets a certain time before it will be asked again, to prevent spamming
+and possibly refused connections. This can be configured by passing a configuration file
+to pythonwhois.set_cool_down_config. This file can contain the following to elements, but doesn't have to.
+`[general]`  
+`default_cool_down_length : 1`  
+This is the general part. Currently, only one variable can be defined. It is optional to do so.
+
+`[whois.eu]`  
+`cool_down_length : 10`  
+`max_requests_minute : 5`  
+`max_requests_hour : 20`  
+`max_requests_day : 50`  
+This is how sections for specific WHOIS servers are defined. The section
+name is the name of the server and the section can contain the listed properties.
+None of them are required. Multiple WHOIS servers can be added to the configuration file.
+
 ## Goals
 
 * 100% coverage of WHOIS formats.
@@ -53,14 +81,6 @@ The manual (including install instructions) can be found in the doc/ directory.
 
 Do note that `ipwhois` does not offer a normalization feature, and does not (yet) come with a command-line tool. Additionally, `ipwhois` is maintained by Philip Hane and not by me; please make sure to file bugs relating to it in the `ipwhois` repository, not in that of `pythonwhois`.
 
-## Important update notes
-
-*2.4.0 and up*: A lot of changes were made to the normalization, and the performance under Python 2.x was significantly improved. The average parsing time under Python 2.7 has dropped by 94% (!), and on my system averages out at 18ms. Performance under Python 3.x is [unchanged](https://github.com/joepie91/python-whois/issues/27). `pythonwhois` will now expand a lot of abbreviations in normalized mode, such as airport codes, ISO country codes, and US/CA/AU state abbreviations. The consequence of this is that the library is now bigger (as it ships a list of these abbreviations). Also note that there *may* be licensing consequences, in particular regarding the airport code database. More information about that can be found below.
-
-*2.3.0 and up*: Python 3 support was fixed. Creation date parsing for contacts was fixed; correct timestamps will now be returned, rather than unformatted ones - if your application relies on the broken variant, you'll need to change your code. Some additional parameters were added to the `net` and `parse` methods to facilitate NIC handle lookups; the defaults are backwards-compatible, and these changes should not have any consequences for your code. Thai WHOIS parsing was implemented, but is a little spotty - data may occasionally be incorrectly split up. Please submit a bug report if you run across any issues.
-
-*2.2.0 and up*: The internal workings of `get_whois_raw` have been changed, to better facilitate parsing of WHOIS data from registries that may return multiple partial matches for a query, such as `whois.verisign-grs.com`. This change means that, by default, `get_whois_raw` will now strip out the part of such a response that does not pertain directly to the requested domain. If your application requires an unmodified raw WHOIS response and is calling `get_whois_raw` directly, you should use the new `never_cut` parameter to keep pythonwhois from doing this post-processing. As this is a potentially breaking behaviour change, the minor version has been bumped.
-
 ## It doesn't work!
 
 * It doesn't work at all?

pythonwhois/__init__.py

@@ -1,11 +1,30 @@
 from . import net, parse
 
+
 def get_whois(domain, normalized=[]):
-	raw_data, server_list = net.get_whois_raw(domain, with_server_list=True)
-	# Unlisted handles will be looked up on the last WHOIS server that was queried. This may be changed to also query
-	# other servers in the future, if it turns out that there are cases where the last WHOIS server in the chain doesn't
-	# actually hold the handle contact details, but another WHOIS server in the chain does.
-	return parse.parse_raw_whois(raw_data, normalized=normalized, never_query_handles=False, handle_server=server_list[-1])
-
-def whois(*args, **kwargs):
-	raise Exception("The whois() method has been replaced by a different method (with a different API), since pythonwhois 2.0. Either install the older pythonwhois 1.2.3, or change your code to use the new API.")
+    final_result = net.get_whois_raw_wrapped(domain, with_server_list=True)
+    # Unlisted handles will be looked up on the last WHOIS server that was queried. This may be changed to also query
+    # other servers in the future, if it turns out that there are cases where the last WHOIS server in the chain doesn't
+    # actually hold the handle contact details, but another WHOIS server in the chain does.
+    if len(final_result.server_list) > 0:
+        handle_server = final_result.server_list[-1]
+    else:
+        handle_server = ""
+    return parse.parse_raw_whois(final_result.responses, normalized=normalized, never_query_handles=False,
+                                 handle_server=handle_server)
+
+
+def set_persistent_cache(path_to_cache):
+    """
+    Set a persistent cache. If the file does not yet exist, it is created.
+    :param path_to_cache: The place where the cache is stored or needs to be created
+    """
+    net.server_cache.set_persistent_location(path_to_cache)
+
+
+def set_cool_down_config(path_to_config):
+    """
+    Set a cool down configuration file, describing specific settings for certain WHOIS servers.
+    :param path_to_config: The path to the configuration file, this needs to exist
+    """
+    net.cool_down_tracker.set_cool_down_config(path_to_config)

pythonwhois/caching/whois_server_cache.py

@@ -0,0 +1,59 @@
+import ast
+import os
+
+
+def read_cache(file_path):
+    if os.path.isfile(file_path):
+        return ast.literal_eval(open(file_path).read())
+    else:
+        if os.path.dirname(file_path):
+            os.makedirs(os.path.dirname(file_path))
+        return {}
+
+
+def write_cache(cache, file_path):
+    cache_file = open(file_path, 'w')
+    cache_file.write(str(cache))
+
+
+class WhoisServerCache:
+    """
+    Cache handler for easy of use. Do not instantiate. import server_cache instead.
+    Otherwise an inconsistent cache can happen as a result of multiple caches.
+    """
+
+    def __init__(self):
+        self.cache = {}
+        self.persistent = False
+        self.file_path = None
+
+    def get_server(self, tld):
+        """
+        Get a WHOIS server for a given TLD
+        :param tld: The TLD to get the WHOIS server for
+        :return: The WHOIS server if it is known, or None otherwise
+        """
+        return self.cache.get(tld)
+
+    def put_server(self, tld, whois_server):
+        """
+        Store a new WHOIS server in the cache. If the cache is persistent,
+        it is also written to disk again. Because the WHOIS servers
+        don't change that often, it simply writes to a file.
+        :param tld: The TLD to store a WHOIS server for
+        :param whois_server: The WHOIS server to store
+        """
+        self.cache[tld] = whois_server
+        if self.file_path is not None:
+            write_cache(self.cache, self.file_path)
+
+    def set_persistent_location(self, file_path):
+        """
+        Store the cache in a persistent location
+        :param file_path: The path to store the cache
+        """
+        self.file_path = file_path
+        self.cache = read_cache(file_path)
+
+
+server_cache = WhoisServerCache()