Release 1.2.12

add version information (#125)

Author: al-niessner
Date: Sat, 22 Feb 2020 14:56:41 -0800

• add version information

Make sure that the versions are always strings by forcing them that way so that the log does not error because of the %s in the log message.

Upgraded test_10 to verify that the log message is correct.

Release 1.2.11

issue 32: aspects implemented similarly (#120)

* make dawgie.Aspects same

Turns out that post and shelf did not have the same key order -- tn, fsvn in post vs. fsvn, tn for shelf. This is a patch to make them the same tn, fsvn.

* fix patch

Need to create the dictionary structure as it is being filled.

Release 1.2.10

Issue 100: enhance rule 01 (#105)

Author: al-niessner
Date: Mon, 27 Jan 2020 12:46:48 -0800

• updated factory comments

The abstract factories are crucial but not well documented. Updated the documentation dawgie to help explain what needs to be done and how it should be done.

• extend rule 01

The factory methods that dawgie depends on were not being fully checked. The number of arguments was being checked but that let errors creep through with defaults and types. Some parts of the system require default values for the factory arguments. Therefore, extended the rule 01 testing to be parameter count first, then default values followed by types.

• updated the testing

use a generator (#104)

Author: al-niessner
Date: Thu, 23 Jan 2020 14:38:50 -0800

• use a generator

as_dict() was a bad shortcut. Moving to creating a generator that returns the tuple that dawgie.db.post does to reflect the use of dawgie.Aspect(). Therefore, removed as_dict() and made the iter() a generator which is supposed to just do the right thing.

use a generator

Author: Al Niessner
Date: Thu, 23 Jan 2020 14:27:38 -0800

Release 1.2.9

Issue 75 improve dawgie.de (#93)

Author: John Engelke
Date: Mon, 13 Jan 2020 10:26:01 -0800

• issue_75: Changes, including fix for bug with duplicate tags, spacing reformat, fix for lists not using 'li', fix to allow 'void' tags proper formatting, addition of capacity to add raw css and stylesheets, addition of pre tag, modify add_primitive to display inline with add_declaration, addition of class and id attributes, spacing tweaks per PEP8, regroup writing of configuration (head) versus presentation (body) tags, disable writing of primitives after declarations since they are now written inline. ...

• issue_75: Changed approach to keep copy of old methods guaranteeing strict backwards compatibility. Also, new method will use inherited __content value to append, then guaranteeing exact ordering with all other subclasses. Touchpoint. ... TO-DO: Modify div for ease-of-use to self-close and include explicit classes and/or style. Also, wrap __content assignments with AsIsText object to ensure they render as expected.

• issue_75: Fixed errant return type and re-added bokeh import removed by IDE during cleanup.

• issue_75: Standardized templating on f-strings, added clean methods to limit malicious code, standardized class, id and style attributes, rearranged order to support writing customizations after file loads for script and css imports, remove superfluous variables added to init, separated legacy and new functions to support backwards compatibility.

• clean up

Keep imports explicit (no from unless name is super long) and keep import alphabetic (or the best that I remember the alphabet)

Co-authored-by: al-niessner [email protected]

Issue 90: excise SDP (#92)

Author: al-niessner
Date: Wed, 8 Jan 2020 10:47:24 -0800

• change dawgie.pl, dawgie.tools, dawgie.fe, and unit testing

Updated farm, start, and state such that all SDP and sdp became FSM and fsm respectively.

• update db modules as well

The shelf and util modules were different then the previeous alterations where SDP was used to rever to an FSM. In this case, sdp was hardcoded as the db_name. Hence, all reference where changed to dawgie.context.db_name.

• update pylint RC

Many of the FSM functions are added at run time by the transistions package. Had to change sdp to fsm in the pylint RC to account for this.

Issue 84: remove hardcoded email alerts (#89)

Author: al-niessner
Date: Wed, 8 Jan 2020 09:51:00 -0800

• add email alert destination

The first part of the problem is to add a configuration or context driven destination for alerts via email. This was done by adding dawgie.context.email_alerts_to that follows the usual paradigm.

• complete email alerts

Updated dawgie.tools.submit to use dawgie.context for the email list. If the list is not defined, then just log it. In fact, the message will always be in the log file for those not on the mailing lists.

Updated dawgie.fe.submit to reflect changes in dawgie.tools.submit.

Issue 76: support https (#79)

Author: al-niessner
Date: Tue, 7 Jan 2020 15:00:59 -0800

• required for twisted to do https

• add hook for cert

Seems that all that is needed is the SSL PEM file as it contains both the cert and the key. Twisted uses this single file to initiate an https server.

• clean up in preparation

Found two mains in dawgie.fe. Removed what appears to be the older one and it looks fine in comparison to the actual main.

• add https hooks

When starting the gui, check to see if a pem file exists. If it does and is a real file, then try to use it as the cert. While this matches the requirements in the twisted documenation it still needs to be tested somehow.

• update rules

twisted.internet.reactor.listenSSL is generated at run time.

• runs standalone with https

The PEM cert and key must be in a single file. Once that was done, it worked.

Issue 87: clean up front-end (#88)

Author: al-niessner
Date: Mon, 6 Jan 2020 12:28:54 -0800

• change to call object

• update DynamicContent

Check self.__fnc to see if it is an instance of dawgie.fe.Defer and act accordingly. Now the call to the object should act like a function too.

• change the function names

Moved the old function names to call so that the object becomes callable.

need to call the function not the object (#86)

Author: al-niessner
Date: Mon, 6 Jan 2020 12:09:35 -0800

Release 1.2.8

issue 80: fix auto merge (#81)

Author: al-niessner
Date: Thu, 2 Jan 2020 14:58:46 -0800

• pass gpghome

Passing the ENV was not enough in this situtation. Instead, need to add the ability to pass the running gpg home to the spawned compliancy check.

• reworked deferred requests

Rendering a state vector was already a deferred task. Often these renderings take a significant amount of time so they are allowed to happen in a separate thread so that the GUI is still responsive. However, it was done in an ad-hoc way that allowed for only it to be done as a deferred. Reworked the concept to allow other tasks to become deferred as well.

Added dawgie.fe.svrender to handler the work of deferring the task to another thread in this module. It now isolates that the deferred in in another thread in this module and how it closes out the work in this module as well.

Updated dawgie.fe.app to import the new module and load the dawgie.fe.DynamicContent with the new methods and objects.

Cleanup up redundant code from dawgie.de.

• refactor code

After pulling out the subprocess.spawn() from dawgie.tools.submit.auto_merge() had to rework the code from the app hook down to become async. This should also free up some part of the GUI response when a submit happens. Not sure though because it may be happening during the reload rather than this portion.

Move the dawgie.pl.start.submit() to dawgie.fe.submit.Process along with the break up of dawgie.tools.submit.auto_merge() in support as well. It is less obvious but is much more async friendly.

issue 82: update the build scripts (#83)

Author: al-niessner
Date: Wed, 1 Jan 2020 16:59:24 -0800

• update the build scripts

• update the license block

emergency fix (#78)

Author: al-niessner
Date: Mon, 16 Dec 2019 11:52:59 -0800
All of the types that could find there way into dawgie.de.factory were not covered. Make it more complete.

Issue_68: Moved modules out of setup.py to requirements.txt for more … (#72)

Author: John Engelke
Date: Thu, 12 Dec 2019 10:19:20 -0800

• Issue_68: Moved modules out of setup.py to requirements.txt for more flexibility. Added method to read in from requirements.txt and feed into setup.py to improve installation flexibility. Fixed error reading README.md when it did not exist in setup directory. Fixed error locating LICENSE.txt due to pathing/naming. Converted file read into Pathlib operation. Converted psycopg2 to install via psycopg2-binary and tested, removing dependency on localized compilers.

• issue_68: Updated code to be more pythonic per suggestions in pull request comments.

• issue_68: Changes per code review to promote pythonic coding.

• issue_68: PEP 8 indentation issue required this to be run through the formatter.

issue_69: Updated GNUPG cleanup method to ignored failed deletes (e.g… (#73)

Author: John Engelke
Date: Wed, 11 Dec 2019 15:09:29 -0800

• issue_69: Updated GNUPG cleanup method to ignored failed deletes (e.g. file does not exist) to compensate for GNU PGP2 creating an intermittent concurrency issue when it cleans up the file itself during the call.

• Remove spacing auto-added to commit previously

issue_15: BASH commands should now be cross-platform. Tested as far as possible such that DAWGIE launched. (#74)

Author: John Engelke
Date: Wed, 11 Dec 2019 12:23:51 -0800

pass current environment (#71)

Author: al-niessner
Date: Wed, 4 Dec 2019 12:20:45 -0800
Between change in external environment which the project is running and possible changes to twisted, need to pass the current environment.

use context.db_rotate (#67)

Author: al-niessner
Date: Mon, 25 Nov 2019 10:53:18 -0800

adjust factories (#65)

Author: al-niessner
Date: Tue, 19 Nov 2019 10:20:01 -0800
dawgie.fe now uses the default display rather than hard coded to dawgie.de.html. This allows projects to build their own visitors since this task is generally tedious and user specific. The current dawgie.de.html is the bare minimum.

dawgie.de.factory() now checks the input for None (default) and uses the module specified in dawgie.context.display. For backward compatibility, if no "." are detected in the module name, it prepends dawgie.de to it. Override with care!

Release 1.2.7

issue 55: test and (#59)

Author: al-niessner
Date: Wed, 6 Nov 2019 14:42:56 -0800

• move some code around

Part of the problem with run ids is that they are controlled from two locations. The simplest control is in dawgie.pl.farm and the other part is on dawgie.pl.scheduler. The first step is to pull the run ID bits from dawgie.pl.farm.dispatch() into their own functions and then test them to make sure they work as expected.

• update testing

test_06.Schedule.test_update() now checks the result of the update and makes sure everyone has the correct run ID. The test, however, is failing because not all of the children are being scheduled. This would explain why tasks have the wrong run IDs. The more complete verification is now that all children are scheduled and that they have the correct run ID.

• actual fixes

Updated dawgie.pl.schedule._priors(). Seems it was checking for dawgie.Task when it should have been dawgie.Algorithm. Fixed this and the test workds just fine.

issue 54: clean out docker (#58)

Author: al-niessner
Date: Mon, 28 Oct 2019 12:47:03 -0700

• remove unused docker

Some docker images are needed for the CI tests (CIT). Left the exercise script as well, but will need to build up a docker image for it at some point.

issue 53: always use the installed version (#57)

Author: al-niessner
Date: Mon, 28 Oct 2019 12:23:25 -0700

Update dawgie.de.html() to insert the appropriate bokeh dependencies based on the installed version of bokeh. It will keep the generated dawgie.StateVector.view() HTML compatible with the installed version of bokeh.

issue 51: patch post2shelve (#56)

Author: al-niessner
Date: Wed, 23 Oct 2019 14:37:48 -0700

• keep run 0 when already 0

Release 1.2.6

Issue 44: fixing up scheduler (#50)

Author: al-niessner
Date: Wed, 16 Oct 2019 11:53:43 -0700

• add capabilities

dawgie.MOMENT added "boot" which will trigger something when the pipeline boots and only boots (not reload from git). This is an odd request but since there is a need, added it.

dawgie.schedule() was updated to properly handle the new "boot" in dawgie.MOMENT. The check was modified to be more adaptable when dawgie.MOMENT changes again in the future.

dawgie.tools.compliant was updated to check that events() returns valid dawgie.MOMENTS since nothing requires the user use dawgie.schedule().

• added unit testing

Single unit test for dawgie.pl.schedule._delay() that it computes 0 delay and then throws the exception when the same unit comes around a second time.

• include more checks on building an EVENT

Issue 45 (#49)

Author: al-niessner
Date: Mon, 14 Oct 2019 18:32:19 -0700

• have a test that highlights the error

• fix bad assumption

dawgie.pl.schedule.update() assumed everything was a dawgie.Algorithm and called previous(). The correction was to use an _priors() function that called traits(), variables(), or previous() depending if it as Analyzer(), Regression(), Task() respectively.

Issue 47: backlog (#48)

Author: al-niessner
Date: Wed, 9 Oct 2019 08:24:23 -0700

• add configurable TCP/IP backlog

As the worker size scales up, the backlog needs to increase with it. Since it should be directly related to the number of workers, called it worker_backlog.

• update all worker interfaces

There are many connections between workers and the pipeline server. Update them all to use a big enough backlog to support the scale of workers being used.

Release 1.2.5

quick fix (#43)

Previous ticket only solved half of the problem. Need to override the value from the command line as well.

Release 1.2.4

issue 31: missing log page (#41)

Author: al-niessner
Date: Wed, 21 Aug 2019 12:24:43 -0700

• update logging

move dawgie.pl.logger.fe from building a complex JSON object using logging.Formatter to building a string using ;\n; as the delimiter between elements. Tests in a notebook show this should work but no test exists for it currently. Will have to test it in a live pipeline at a later date.

updated dawgie.fe.app to convert the now [{}] returned by dawgie.pl.logger.fe.remembered() to a JSON object using json.dumps(). This should prevent further problems with the object being invalid in the future.

Verifiying the display order matches the order from the json. I was reverse ordering the logs, but no longer. (#40)

Author: Jacob Llamas
Date: Tue, 20 Aug 2019 16:30:14 -0700
Updated the logs function to show new entries at the top instead of the bottom.

Issue 36 QoL update (#39)

Author: Jacob Llamas
Date: Tue, 20 Aug 2019 14:55:15 -0700

• Fixing a bug where filtering was not working for State Vector searches.

• Hiding header rows on SV searches when filtering.

Updating site config to not flash {{tag_names}} before replacing them with content.

• Fixing a js bug in branding, when an element does not exist.

Fixing a bug where filtering was not working for State Vector searches. (#38)

Author: Jacob Llamas
Date: Tue, 20 Aug 2019 08:47:49 -0700

Issue 29: filter defaults (#34)

Author: al-niessner
Date: Wed, 7 Aug 2019 16:11:58 -0700

• non-sequitur changes

Added some logging to help understand why navel gazing is taking so long. They were needed to debug the private pipeline to the point were this issue could be tested.

• Improper handling of object or string

dawgie.fe.app.search_filter_{a,d,u}() were not uniformly handling the type of the object. Some parts treated it as a dictionary others a string. Made it a dictionary consistently until it is turned to JSON at the return.

Issue 20: correct substring (#33)

Author: al-niessner
Date: Tue, 6 Aug 2019 17:13:27 -0700

• new requirements becuase actual deploy is done via PyPI now but local tools like .ci/exercise_01.sh still require the docker image

• since the targets are sorted, ignore the all

Issue 27: expand compliant checks (#28)

Author: al-niessner
Date: Fri, 2 Aug 2019 12:56:37 -0700

• add rule 9: no state vectors

First, add the rule_09() to tools/compliant.py. It simply runs through the factories and makes sure that all units return an non-zero length array.

Second, add a negative test to Test/test_08.py along with a bad AE (bae). Since there is a positive test already to verify Test/ae will work, the negative test will verify that rule 9 is coded correctly.

Lastly, using the test with the bae correct the code and test until it all works.

• non sequitour change

Updated dawgie.tools.dag to have a verbose switch. That is when it was discovered that state_vectors() must not be zero length to get added to the dag. Still, these changes would be nice to expand upon in the future so keeping them here.

Release 1.2.3

niessner@elysium:~/Projects/DAWGIE$ cat /tmp/t.t

issue 25: PyPI script (#26)

Author: al-niessner
Date: Sun, 28 Jul 2019 16:19:21 -0700

• .ci/pypi.sh

While not part of continuous integration, it is a helper script for me to take a release, build it, then install it at PyPI for those using DAWGIE.

Issue 23: fix up setup.py (#24)

Author: al-niessner
Date: Sun, 28 Jul 2019 15:57:52 -0700

• add a little more stuff to delivery and try using a patched version of pydot3.

Added README.md and LICENSE to the egg data directory just because. Seems better to have it all around than to not have it at all. Might be misguided.

Built a patched version of pydot3 in my fork of it then pushed it to PyPI as dawgie-pydot3. Once built and deployed, then changed this setup.py to depend upon it.

• fix vulnerability

Issue 21: start PyPI (#22)

Author: al-niessner
Date: Sat, 27 Jul 2019 13:44:57 -0700

• update setup.py

Following the PyPI documentation, try building the package for distribution on PyPI. Will need a parent script to do more work than this.

• start of deploying via PyPI

An instiall script to build the a deployment. Right now the script builds this directory rather than collecting a released version from github.com. It will come with time, but here the mechanics of how to build the artifact for PyPI and deliver it is complete. The script is left in safe mode where it will deploy to test.pypi instead of pypi.

Updated the setup.py to work correctly with PyPI. It does a description (poorly) but does one. It has a license and describes it nicely on the website. All inall, it looks like the setup.py is sufficient.

update utility for making release notes

Author: Al Niessner
Date: Tue, 23 Jul 2019 14:02:29 -0700