Create WAF traversal function

[rshewitt]

User Story

In order to collect files from a WAF, datagov wants create a function in the harvesting logic repo which can traverse a WAF and pull all the files.

Acceptance Criteria

• GIVEN a WAF
  WHEN the function is invoked
  THEN all the files in the tree are collected and downloaded.

Background

[Any helpful contextual notes or links to artifacts/evidence, if needed]

Security Considerations (required)

[Any security concerns that might be implicated in the change. "None" is OK, just be explicit here!]

Sketch

def traverse_waf( url, files=[], to_find=".xml' ):
  # resp = requests.get( url ) 
  # bs4 parse response 
  # get me all the "to_find" files and store in "files"
  # if you find a folder then traverse_waf( folder_url, files ) 

  return files

def download_files( files ):
  for file in files:
    #download the file

[rshewitt]

draft pr

[jbrown-xentity]

Not that we want to do it the way it's currently done necessarily, but the current code lives here: https://github.com/ckan/ckanext-spatial/blob/master/ckanext/spatial/harvesters/waf.py#L52

[jbrown-xentity]

Watch out for looping; many WAFs have a folder structure that allows you to go "up" a level; if you don't exclude those you can end up in an indefinite search. It doesn't matter if we do a Breadth First Search or a Depth First Search, but we do need to be able to look at sub-folders and list/download them as well.

[rshewitt]

i'm checking for the parent here. the pr is a work in progress. i included a filter list to the function as a way to exclude directories we don't want to open which could be source specific.