updated v0.5.3

- Updated order status.
- Check existing files before attempting redirect url and download.
- Overall optimization for faster check and updated readme for fallback bundles.

README.md

@@ -157,7 +157,7 @@ Or you can use [anaconda to install](https://conda-forge.github.io/). Again, bot
 
 Make sure you initialized planet client by typing ```planet init```. As usual, to print help:
 
-![porder_main](https://user-images.githubusercontent.com/6677629/61379317-4ee85600-a875-11e9-9234-5f57dccf9588.png)
+![porder_main](https://user-images.githubusercontent.com/6677629/68093060-a3810280-fe5f-11e9-8dbd-b5f59d42ea49.png)
 
 To obtain help for a specific functionality, simply call it with _help_ switch, e.g.: `porder idlist -h`. If you didn't install porder, then you can run it just by going to *porder* directory and running `python porder.py [arguments go here]`
 
@@ -257,7 +257,13 @@ porder bundles --item "PSScene4Band"
 ```
 
 ### order
-This tool allows you to actually place the order using the idlist that you created earlier. the ```--op``` argument allows you to take operations, delivery and notifications in a sequence for example ```--op toar clip email``` performs Top of Atmospheric reflectance, followed by clipping to your geometry and send you an email notification once the order has completed, failed or had any any change of status. An important changes is the concept of passing bundles instead of using assets. The list of operations for the ```--op``` are below and ** the order of these operations is important**
+This tool allows you to actually place the order using the idlist that you created earlier. the ```--op``` argument allows you to take operations, delivery and notifications in a sequence for example ```--op toar clip email``` performs Top of Atmospheric reflectance, followed by clipping to your geometry and send you an email notification once the order has completed, failed or had any any change of status. An important changes is the concept of passing bundles instead of using assets. Bundles are predefined meaning all assets in a bundle are not available for an item your attempt at downloading that attempt will fail.
+
+For example if an item id '20181227_125554_0f4c' does not have surface reflectance asset type. So if you try to download this using bundle type analytic_sr_udm2 it will not work, similary if you order an item where a specific operation cannot be performed for example if you order visual and then try to do bandmath with four bands. These examples and more are where **fallback bundles** come in handy. Think of this as providing a list of bundles to keep trying if one bundle type fails. The priority goes left to right. You can provide comma seperated fallback bundles for example as
+
+```analytic_sr_udm2,analytic``` instead of ```analytic_sr_udm2``` to avoid certain items from failing to download.
+
+The list of operations for the ```--op``` are below and ** the order of these operations is important**
 
 clip|toar|comp
                         osite|zip|zipall|compression|projection|kernel|aws|azu
@@ -369,6 +375,11 @@ A simple setup would be
 
 ## Changelog
 
+### v0.5.3
+- Updated order status.
+- Check existing files before attempting redirect url and download.
+- Overall optimization for faster check and updated readme for fallback bundles.
+
 ### v0.5.2
 - Added harmonization tool to harmonize PS2.SD to PS2.
 - Improvements and error handling to quota tool

porder/__init__.py

@@ -2,4 +2,4 @@
 
 __author__ = 'Samapriya Roy'
 __email__ = '[email protected]'
-__version__ = '0.5.2'
+__version__ = '0.5.3'

porder/async_downloader.py

@@ -24,10 +24,13 @@
 import os
 import sys
 import csv
+import glob
 from retrying import retry
 from pySmartDL import SmartDL
 from planet.api.utils import read_planet_json
 from planet.api.auth import find_api_key
+from datetime import datetime
+
 
 #Get Planet API and Authenticate SESSION
 try:
@@ -88,48 +91,67 @@ def downonly(redirect_url,local_path,ext,items,ulength):
             print("Checking "+str(ulength-i)+" remaining ==> "+"File already exists SKIPPING: "+str(os.path.split(local_path)[-1]))
             i=i+1
 
-
-# Get the redirects and download
-def asyncdownload(url,local,ext):
+def get_session_response(url):
     response=SESSION.get(url).json()
     print("Polling ...")
     while response['state']=='queued' or response['state']=='running' or response['state']=='starting':
         bar = progressbar.ProgressBar()
-        for z in bar(range(60)):
+        for _ in bar(range(60)):
             time.sleep(1)
         response=SESSION.get(url).json()
+
+    return response
+
+# Get the redirects and download
+def asyncdownload(url,local,ext):
+    filenames = glob.glob1(local,'*')
+    response = get_session_response(url)
     if response['state']=='success' or response['state']=='partial':
         print('Order completed with status: '+str(response['state']))
-        ulength=len(response['_links']['results'])
-        for items in response['_links']['results']:
+        ulength=len(response['_links']['results'])-len(filenames)
+        for index, items in enumerate(response['_links']['results']):
+            expiration_time = datetime.strptime(items['expires_at'], "%Y-%m-%dT%H:%M:%S.%fZ")
+            if datetime.now() > expiration_time:
+                print('The URL links have expired.  Refreshing.')
+                response[:] = get_session_response(url)
+                if response['state']=='success' or response['state']=='partial':
+                    items = response['_links']['results'][index]
+                else:
+                    print('Order Failed with state: '+str(response['state']))
+                    return
             url=(items['location'])
             name=(items['name'])
-            url_to_check = url if url.startswith('https') else "http://%s" % url
-            redirect_url = check_for_redirects(url_to_check)
-            if redirect_url.startswith('https'):
-                if name.endswith('manifest.json'):
-                    time.sleep(0.2)
-                    resp=SESSION.get(url)
-                    if int(resp.status_code)==200:
-                        r=resp.content
-                        inp=json.loads(r)
-                        for things in inp['files']:
-                            try:
-                                local_path=os.path.join(local,things['annotations']['planet/item_id']+'_manifest.json')
-                            except Exception as e:
-                                local_path=os.path.join(local,things['path'].split('/')[1].split('.')[0]+'_manifest.json')
-                    else:
-                        print(resp.status_code)
+            if name.endswith('manifest.json'):
+                time.sleep(0.2)
+                resp=SESSION.get(url)
+                if int(resp.status_code)==200:
+                    r=resp.content
+                    inp=json.loads(r)
+                    for things in inp['files']:
+                        try:
+                            local_path=os.path.join(local,things['annotations']['planet/item_id']+'_manifest.json')
+                        except Exception as e:
+                            local_path=os.path.join(local,things['path'].split('/')[1].split('.')[0]+'_manifest.json')
                 else:
-                    local_path=os.path.join(local,str(os.path.split(items['name'])[-1]))
-                try:
-                    downonly(redirect_url,local_path,ext,items,ulength)
-                except Exception as e:
-                    print(e)
-                except (KeyboardInterrupt, SystemExit) as e:
-                    print('\n'+'Program escaped by User')
-                    sys.exit()
+                    print(resp.status_code)
+            else:
+                local_path=os.path.join(local,str(os.path.split(items['name'])[-1]))
+            filenames=[os.path.join(local,files) for files in filenames]
+            if not local_path in filenames:
+                url_to_check = url if url.startswith('https') else "http://%s" % url
+                redirect_url = check_for_redirects(url_to_check)
+                if redirect_url.startswith('https'):
+                        try:
+                            downonly(redirect_url,local_path,ext,items,ulength)
+                        except Exception as e:
+                            print(e)
+                        except (KeyboardInterrupt, SystemExit) as e:
+                            print('\n'+'Program escaped by User')
+                            sys.exit()
+            else:
+                print("File already exists SKIPPING: "+str(local_path))
     else:
         print('Order Failed with state: '+str(response['state']))
+
 ##download(url="https://api.planet.com/compute/ops/orders/v2/0ee9e923-59fc-4c31-8632-9882cb342708",
 ##         local=r"C:\planet_demo\terra",errorlog=r"C:\planet_demo\errorlog.csv")

porder/downloader.py

@@ -22,6 +22,7 @@
 import time
 import progressbar
 import json
+import glob
 import os
 import sys
 from retrying import retry
@@ -105,10 +106,11 @@ def get_session_response(url):
     return response
 
 def download(url,local,ext):
+    filenames = glob.glob1(local,'*')
     response = get_session_response(url)
     if response['state']=='success' or response['state']=='partial':
         print('Order completed with status: '+str(response['state']))
-        ulength=len(response['_links']['results'])
+        ulength=len(response['_links']['results'])-len(filenames)
         for index, items in enumerate(response['_links']['results']):
             expiration_time = datetime.strptime(items['expires_at'], "%Y-%m-%dT%H:%M:%S.%fZ")
             if datetime.now() > expiration_time:
@@ -121,31 +123,35 @@ def download(url,local,ext):
                     return
             url=(items['location'])
             name=(items['name'])
-            url_to_check = url if url.startswith('https') else "http://%s" % url
-            redirect_url = check_for_redirects(url_to_check)
-            if redirect_url.startswith('https'):
-                if name.endswith('manifest.json'):
-                    time.sleep(0.2)
-                    resp=SESSION.get(url)
-                    if int(resp.status_code)==200:
-                        r=resp.content
-                        inp=json.loads(r)
-                        for things in inp['files']:
-                            try:
-                                local_path=os.path.join(local,things['annotations']['planet/item_id']+'_manifest.json')
-                            except Exception as e:
-                                local_path=os.path.join(local,things['path'].split('/')[1].split('.')[0]+'_manifest.json')
-                    else:
-                        print(resp.status_code)
+            if name.endswith('manifest.json'):
+                time.sleep(0.2)
+                resp=SESSION.get(url)
+                if int(resp.status_code)==200:
+                    r=resp.content
+                    inp=json.loads(r)
+                    for things in inp['files']:
+                        try:
+                            local_path=os.path.join(local,things['annotations']['planet/item_id']+'_manifest.json')
+                        except Exception as e:
+                            local_path=os.path.join(local,things['path'].split('/')[1].split('.')[0]+'_manifest.json')
                 else:
-                    local_path=os.path.join(local,str(os.path.split(items['name'])[-1]))
-                try:
-                    downonly(redirect_url,local_path,ext,items,ulength)
-                except Exception as e:
-                    print(e)
-                except (KeyboardInterrupt, SystemExit) as e:
-                    print('\n'+'Program escaped by User')
-                    sys.exit()
+                    print(resp.status_code)
+            else:
+                local_path=os.path.join(local,str(os.path.split(items['name'])[-1]))
+            filenames=[os.path.join(local,files) for files in filenames]
+            if not local_path in filenames:
+                url_to_check = url if url.startswith('https') else "http://%s" % url
+                redirect_url = check_for_redirects(url_to_check)
+                if redirect_url.startswith('https'):
+                        try:
+                            downonly(redirect_url,local_path,ext,items,ulength)
+                        except Exception as e:
+                            print(e)
+                        except (KeyboardInterrupt, SystemExit) as e:
+                            print('\n'+'Program escaped by User')
+                            sys.exit()
+            else:
+                print("File already exists SKIPPING: "+str(local_path))
     else:
         print('Order Failed with state: '+str(response['state']))
 # download(url="https://api.planet.com/compute/ops/orders/v2/0ee9e923-59fc-4c31-8632-9882cb342708",

porder/order_size.py

@@ -56,7 +56,7 @@ def parsesize(url):
 def ordersize(url):
     response=SESSION.get(url).json()
     print("Polling ...")
-    while response['state']=='running' or response['state']=='starting':
+    while response['state']=='queued' or response['state']=='running' or response['state']=='starting':
         bar = progressbar.ProgressBar()
         for z in bar(range(60)):
             time.sleep(1)

setup.py

@@ -20,7 +20,7 @@ def readme():
         return f.read()
 setuptools.setup(
     name='porder',
-    version='0.5.2',
+    version='0.5.3',
     packages=['porder'],
     url='https://github.com/samapriya/porder',
     package_data={'': ['bundles.json']},