Release 1.4.4

Issue 245: add a reset (#252)

* implement reset

Make it a post with no parameters. Not sure why that is any better than a GET but seems like accidental GET are easy while accidental POST with no parameters is more rare. Probably not in reality, but seems like something to make one sleep easier.

Release 1.4.3

What's Changed

• Issue 226: set to unique fifo by @al-niessner in #227
• Update config.yml by @al-niessner in #228
• Update Dockerfile.py by @al-niessner in #229
• 230: quick fix by @al-niessner in #231
• 232: sytle can be blank by @al-niessner in #233
• content fix by @al-niessner in #234
• 236: fix logging problem by @al-niessner in #238
• 235: add wall time by @al-niessner in #239
• 237: ignore trace for now by @al-niessner in #240
• 241: inter improvements by @al-niessner in #242
• 243: add __repr__ to classes by @al-niessner in #244
• Issue 208: move to TLS by @al-niessner in #246
• Issue 248: complete exercise by @al-niessner in #249

Full Changelog: 1.4.2...1.4.3

Release 1.4.2

What's Changed

• Update model.py by @gbryden in #217
• Update post2shelve.py by @al-niessner in #218
• improve snapshot by @al-niessner in #220
• Issue 213: make db.reset() explicit by @al-niessner in #222
• 145: Fixes to promotion by @al-niessner in #223

New Contributors

• @gbryden made their first contribution in #217

Full Changelog: 1.4.1...1.4.2

Release 1.4.1

214: problem with shelve comm link (#215)

Author: al-niessner
Date: Tue, 22 Aug 2023 13:18:12 -0700

• cleaned up shelve

Added a unit test for the work that could be done with mock items. Then did some debugging in-situ with a live pipeline and made other fixes that are missed with the mock items.

issue 199: verify promotion and shelve work in real life (#212)

Author: al-niessner
Date: Wed, 2 Aug 2023 21:32:16 -0700

• clean up

Missed a skip unit test that flushed out a couple more fixes to shelve. There is also a hidden function in the db API that needs to be fixed. It is noted on #213

fixes for it to work (#211)

Author: al-niessner
Date: Fri, 28 Jul 2023 17:50:18 -0700

Issue 206: fix foreman layoff (#209)

Author: al-niessner
Date: Thu, 27 Jul 2023 18:39:19 -0700

• allow dispatch to que jobs for later

Issue 206: scheduler/farm/workers not getting things done (#207)

Author: al-niessner
Date: Mon, 17 Jul 2023 08:34:58 -0700

• allow chaining of check without network

The CI tests always failed if it could not update github. Changed that to fail on report content so can be chained nicely on the CLI without a network connection to github.com.

• add a snapshot capability

added a snapshot module to the dawgie.pl to build a dictionary that represents that moment in time. It grabs internal state from dawgie.context, dawgie.pl.state (fsm), dawgie.pl.farm, and dawgie.pl.schedule. It is really only helpful for debugging. Tested that items could be JSON serialized and the appear to be. Ready to move to ops for needed testing.

The updates include the implementation of the new endpoint /app/snapshot.

update comments

Author: Al Niessner
Date: Wed, 28 Jun 2023 14:20:21 -0700

Release 1.4.0

Issue 194: updates to scheduler and testing of it (#205)

Author: al-niessner
Date: Wed, 28 Jun 2023 13:11:38 -0700

• first steps in fix

Improved documentation so the next time this code is looked at or reviewed,
it can go faster. It works out since what it does is no longer in my head and
this documentation represents what I found difficult to decipher.

• make all test runnable independently

Having to wait for all the tests to run was slowing down development. Wrapped it into this branch because this work is required sooner rather than later.

• fixed previous test that caused the problem

Turns out that promotion engine tests set something that caused this test to fail. Implemented a tearDownClass() to undo what it set. Fixed the tests when they all run.

add default value to help (#202)

Author: al-niessner
Date: Wed, 3 May 2023 20:51:19 -0700
Co-authored-by: Al Niessner [email protected]

stronger check for true (#201)

Author: al-niessner
Date: Wed, 3 May 2023 17:29:35 -0700

add more context to the error message (#200)

Author: al-niessner
Date: Wed, 3 May 2023 16:12:16 -0700

Issue 189: redo shelf to shelve (#193)

Author: al-niessner
Date: Wed, 3 May 2023 15:45:29 -0700

• rewriting shelf

Changed a bunch of names (not all) from shelf to shelve while doing the rewrite just to make it match. It makes it much nicer in understanding the mapping of databases.

Cleaned up post2shelve to use the new shelve tables that account for versions.

Still had a bit of shelf laying about that testing found. Cleaned it up and changed some of the test harness to fit the new shelve.

The bulk of the changes came when the unit tests started running. It was here that most of the type changes and the whole concept of the db changes worked itself out.

renamed redirect to retarget (#192)

Author: al-niessner
Date: Mon, 19 Dec 2022 08:43:59 -0800

• renamed redirect to retarget

Fixed #190 as well since it was in the same location and easy.

Renamed the functions and checked that all connections still work. However, updating the past items still needs to be done.

• retarget implementation for postgres

Did the simplest/hardest all at once. Create duplicate target entries for upstream items by simply walking the prime table. It is a little lazy and it does the latest rid for a task and algorithm. If state vectors and values are well behaved as in defined at algorithm instantiation like they are supposed to be then this shortcut is sufficient.

Leaving shelf for when it is made to really behave like postgres.

• informational updates

Cleaned up error and log message. Changed the look of retarget() names.

Update init.py

Author: al-niessner
Date: Thu, 13 Oct 2022 14:01:57 -0700

issue_186: maybe fix timelines (#188)

Author: al-niessner
Date: Thu, 13 Oct 2022 13:14:05 -0700

• maybe fix timelines

Could not exactly replicate the error in the log file but found a simple error and repaired it. Given how algorithm and sv PKs work, it is necessary to allow multiples and then work with the latest in the database (largest primary table PK).

• fixed timelines

The notebook page was running with the older psycopg2 which worked. Moved to the correct psycopg (3.1.9) and duplicated the error. Corrected it code using fetchall() as it now returns a list of tuples.

Issue 185: infinite recursion (#187)

Author: al-niessner
Date: Fri, 7 Oct 2022 14:34:29 -0700

• change to show problem

• fix for infinite recursion

Issue 183: extend interface (#184)

Author: al-niessner
Date: Fri, 7 Oct 2022 12:37:16 -0700

• add two helper functions to the API

It should be simple to add new targets. It is simple to use an Analyzer to compute all of the desireable targets. It is not easy to have that target added to the database without doing some dawgie gymnastics. Added dawgie.db.add() to remove these gymnastics from the AE.

The idea of sub-targets has come up. In the view of the AE some targets may have sub processing channels that are better shown with sub-target names like A(b) where A is the primary target and b is the sub-target. In this sense, algorihm foo() now works on sub targets A(b) and A(c) independently and in parallel rather than sequentially.

• add two helper functions to the API

It should be simple to add new targets. It is simple to use an Analyzer to compute all of the desireable targets. It is not easy to have that target added to the database without doing some dawgie gymnastics. Added dawgie.db.add() to remove these gymnastics from the AE.

The idea of sub-targets has come up. In the view of the AE some targets may have sub processing channels that are better shown with sub-target names like A(b) where A is the primary target and b is the sub-target. In this sense, algorihm foo() now works on sub targets A(b) and A(c) independently and in parallel rather than sequentially.

• update testing

The change in dawgie.db requires testing to show that any implementation for it works as expected.

• implementation of dawgie.db.add()

Post dawgie.db.post and dawgie.db.shelf required updating. Both have implementations and the postgres has been tested.

repair dawgie.db.post.trace() (#182)

Author: al-niessner
Date: Fri, 19 Aug 2022 16:47:49 -0700
The original code simply found the max for each of the different version values (design, impl, bugfix). The function needs to compute the max in a cascading for such that bugfix.max() is constrained by impl.max() which is then constrained by design.max() so that 1.1.7, 1.9.3 and 2.4.1 returns 2.4.1 and not 2.9.7. The latter is what the prefix code was doing and now the former is correctly returned with these changes.

handle as info() not error() (#180)

Author: al-niessner
Date: Fri, 19 Aug 2022 16:21:23 -0700
Changed the logging from exception() to info() because these are allowed exceptions that allow a success status despite no output being generated. It means there is a data error but nothing that a stack trace helps with at least.

Release 1.3.3

issue 176: find actual cause (#177)

Author: al-niessner
Date: Thu, 18 Aug 2022 15:55:58 -0700

• fix two bugs

.ci/Dockerfile.py needs to use latest pydot that keeps up pyparsing changes

Test/test_10.py fixes a hidden dependency that test 2 ran already and set dawgie.context.fsm.

• cleaned up some testing

Seems with latest ubuntu and dawgie dependencies, some tests needed cleaned up after updating pydot.

Wrote a report for the compliance testing (check 5) too.

• update to latest psycopg

While it appears not to be necessary, updated to latest psycopg because psycopg2 seems to be at the end of it development. It did not fix the underlying problem with test 13 not did upgrading to latest postgres. Still, the changes are in a positive direction so keeping them.

• testing and db interface fixed

Seems there where some hidden problems with the db but not sure what caused the change to make them break suddnely.

Update the testdata.py because one of the string formats whas changed but forgot the f at the front of the string. That fixed all of the strange reductions in keys and such.

Update the post.py because either psycopg and/or psycopg2 both changed and what is returned is different than what it used to be. Anyway, all of test_13 is passing again.

• found the root cause (finally)

Can transition to Archiving from either Updating or Running. If from Updating, then the database is close. If from Running the database is open. Since Archiving needs the database open for the dawgie.tools.trace to run but closed for dawgie.db.archive(), Archiving needed to manage the open state of the database. Upgraded dawgie.db.reopen() to hint at its state when reopen() is called. This allows Archiving to set the database open state appropriately upon completion. This too was tricky because the trigger to exit Archving is called in _archive_done() not in _archive(). Therefore resetting the database open state has to be done in _archive_done() to avoid a race condition.

Update post2shelve.py (#179)

Author: al-niessner
Date: Thu, 11 Aug 2022 13:21:32 -0700

Release 1.3.2

restructure (#175)

Author: al-niessner
Date: Tue, 2 Aug 2022 13:38:10 -0700
Part of the clean up was to make all of the db function fail if open had not been called first. Modified dawgie.tools.trace to open and close the db always since the pipeline has correctly closed it already.

changed order of operations (#173)

Author: al-niessner
Date: Sun, 31 Jul 2022 17:34:39 -0700
Quick notebook test shows that changing inserting every value into shelf vs uypdating a shelf is 3x or so slower. Updated the algorithm to build an in-memory dictionary then update the shelve once done making the dictionary.

info/debug difference

Author: Al Niessner
Date: Sun, 31 Jul 2022 14:44:34 -0700

simply type (#171)

Author: al-niessner
Date: Sun, 31 Jul 2022 14:03:26 -0700

Release 1.3.1

quick fix (#168)

Author: al-niessner
Date: Thu, 28 Jul 2022 12:28:13 -0700

fix typo

Author: Al Niessner
Date: Thu, 28 Jul 2022 11:13:45 -0700

make them match at warning (#166)

Author: al-niessner
Date: Thu, 28 Jul 2022 09:28:19 -0700

simple fix (#165)

Author: al-niessner
Date: Thu, 28 Jul 2022 09:21:25 -0700

Issue 160: update base OS and all tests (#161)

Author: al-niessner
Date: Wed, 29 Jun 2022 10:09:59 -0700

• docker images build

Updating to Ubuntu 22.04 required newer versions of many of the tests like pylint as well. These changes required a bit tinkering to get all tools at a level where they are happy to work together.

• pylint improvements

The bulk of the upgrade goes to pylint. It has improved its checking depth and breadth. Code updates were required to clear the new and improved checks while some inline comments were more appropriate to the design of DAWGIE.

• remove some duplication

One of the newer checks is for code duplication. Other than the __main__ block, duplicate blocks were removed and code cleaned up. Used an inline comment for the main blocks because they are naturally going be the same code over and over. The code is short and concise and really does not lend itself well to be being removed.

Issue 140: fix envs (#159)

Author: al-niessner
Date: Fri, 25 Mar 2022 14:17:08 -0700

• updated all envs and references

Went through dawgie.context and found all the environment variables. Found all references using find. Added DAWGIE_ as prefix.

Release 1.3.0

Issue 139: implement promotion engine (#144)

Author: al-niessner
Date: Sun, 20 Feb 2022 15:55:33 -0800

• finer control over promotion

Added the switch dawgie.context.allow_promotion that can turn it on and off easily. There are conditions that if not followed will really foul up promotion. The belief is that EXCALIBUR might currently be existing in those conditions such that promotion will be bad for them.

• extend unittest

The unit testing PromotionEngine required extension to properly test the addition of dawgie.context.allow_promotion.

• extend promotion

dawgie.pl.promotion to use the current AE to find the dependencies.

dawgie.pl.context changed to support the new allow_promition.

updated test 12 to handle the allow_promotion flag with if the AE is set.

• skeletal promotion

• mark this point

It seems this is mostly wrong but want to mark this point in case the new idea is wrong and need to go back. The changes will remove the constraint first and make it based on just he output and what it will require to run. If it has been run with those conditions previously then use just the already known output. Does not require the complicated constraints and inputs that this edition has partially impleemted.

• promotion should find junctures

Worked consistent() a couple of times to finally arrive at this algorithm should select the most recent equivalent output that matches the current inputs. Left comments in place to highlight the algorithm. It bases its decision on the current version of the algorihm and its output over time. Using the input it would have now if it ran, it then uses the output runids to check for inputs that match what would be the lastest inputs.

The most interesting point, is that promotion should go and this should be done always for all items. However, shelf cannot currently support this level of interigation. However, it still should be scheduled for the next big releaes of dawgie to replace the current scheduler.

• promition adds entries

Now that dawgie.db.consistent() returns what should be a valid juncture, check then install the juncture. promotion() was updated to reflect a possible failure of the juncture insert will force the item to run.

All checks pass but unit testing is not done.

• update all files for license/copyright updates

This update is a pain and probably unnecessary but it is nice in that forces an update to containers, external dependencies, etc that can be easily forgotten. In updating the Dockerfile.cit moved to more modern tools that worked with a newer Ubuntu 20.04 in Dockerifle.os. Many other changes took place after these, but they were the start.

• .ci/check_03.sh updated for new tests

Test/test_13.py was added to do the bulk of the promotion testing within the databases themselves. Right now, shelf is not able to support promotion so postgres is the only database being tested. Test/test_13.py will also work well to have consistent behavior among all supported databases.

To support Test/test_13.py, .ci/check_03.sh was extended to download postgres image, start it, and prepare it (createdb) then clean up the container when finished.

• logging seems to have changed

Somehow, maybe pytest, logging.basicConfig() is being called and so the handler is no longer being set. Changed to add the handler to the child used for testng. Since the main probgram has more control over logging.basicConfig() and issue 45 focused on the handlers behavior, this change so keep the spirit of the test while ignoring updates to external tools.

• expand check_03,sh setup

Now that promotion can be checked for postgres, it is important to load and install postgres during check_03.sh. Since it uses the postgres:latest docker image to complete this work, the expansion of check_03.sh is direct. It first checks to see if the docker image is present then the docker network then it starts the container and creates the needed database. When complete, it stops the docker container.

• apparently bridge is not good so default to citnet

Co-authored-by: Al Niessner [email protected]

add exercise build

Author: Al Niessner
Date: Tue, 7 Dec 2021 14:51:43 -0800

Issue #153 : scripts to help inter data (#154)

Author: al-niessner
Date: Sat, 13 Nov 2021 08:29:07 -0800

• tool to inter data for a mausoleum

It currently only works with postgres and only processes the backup file to find and isolate the desired state vectors.

• move blovs too

• remove unnecessary dirs

• add licensing info

Co-authored-by: Al Niessner [email protected]
Co-authored-by: Al Niessner [email protected]

Issue 156: move to circle CI (#157)

Author: al-niessner
Date: Sat, 13 Nov 2021 08:07:13 -0800

• move from travis to circle

The Yaml and concept between circle and travis is completely different. Updated environment variables ans the like. Created a new GHE token with the correct access and it works.

issue #151 - fix first param (#152)

Author: al-niessner
Date: Mon, 29 Mar 2021 17:53:00 -0700

• fix first param

In each case, the first param (name) should be a string.

• update test code

Declared the first parameter of every factory function a string

Issue #149 - fix dawgie.tools.compliant (#150)

Author: al-niessner
Date: Mon, 29 Mar 2021 17:20:22 -0700

• use inspect correctly

The old code, for whatever inane reason, used str(param) then did string work. Instead use param.default and param.annotation to extract the information correctly. Somewhere between this version of Python and 3.8.5 the str(param) is going to add spaces breaking the code.

bug fixes from 1.2.x branch (#148)

Author: al-niessner
Date: Thu, 3 Dec 2020 09:51:58 -0800

• add version information (#125)

• add version information

Make sure that the versions are always strings by forcing them that way so that the log does not error because of the %s in the log message.

Upgraded test_10 to verify that the log message is correct.

• fix security problem (#147)

Issue 136: add promotion hooks (#138)

Author: al-niessner
Date: Fri, 1 May 2020 12:42:02 -0700

• alter scheduler to enable auto promotion

Added hooks to dawgie.pl.schedule to trigger the auto promotion. Insterted the call to the dawgie.pl.schedule.update() which will add items as required by results sent from dawgie.pl.worker. Added a conditional to dawgie.pl.schedule.next_batch() that will do promotion before scheduling new jobs so that the new jobs have the current best estimate to work with.

Added dawgie.pl.auto which contains the stub for Promote that keeps the current behavior but concentrates where changes need to be made for auto promotion.

• add unittest

While there is not much to test, add what can be done. The first and basically only interesting test is to see that call is working as expected. Wrong ars are processed correctly as is more. Log message shows up correctly as well.

• rename it all

Unhappy with dawgie.pl.auto.Promote renamed it to dawgie.pl.promotion.Engine.

Co-authored-by: Al Niessner [email protected]

update new values (#135)

Author: al-niessner
Date: Mon, 20 Apr 2020 12:09:30 -0700

• update new values

dawgie.Analysis, dawgie.Regress, and dawgie.Task are all updated to expect a tuple of name,isnew instead of just name. dawgie.pl.worker already handles the list as an object and simply pass it along requiring no changes.

dawgie.db.post and dawgie.db.shelf update the list and are updated as well. Instead of adding the list conditionally, they always add to the list and pass along if it is new or not.

dawgie.pl.schedule was updated to work with the tuple. It behaves as it always has or at least that is the intent but now allows for expansion in ability.

• update testing

The test used just a list of strings updated it to be a list of tuples.

Co-authored-by: Al Niessner [email protected]

Issue 60: leaving state vector empty (#133)

Author: al-niessner
Date: Mon, 13 Apr 2020 13:22:58 -0700

• handle invalid data exceptions

dawgie supports two types of exceptions to be thrown: dawgie.NoValidInputDataError and dawgie.NoValidOutputDataError. In these two cases, the job should be considered a success but not move the target forward in the processing tree. This creates a try state condition which will be True (full success), None (invalid data exception), False (all other exceptions).

• handle the tri-state

Add an invalid state to the messaging that is passed about. Updated dawgie.pl.farm to translate the tri-state message from teh workers to the dawgie.pl.jobinfo.State.{failure,invalid,success} as required.

• connect the error step

Added dawgie.pl.schedule.purge() with empty implementation. Walking down the tree and removing targets is going to take some work. Just put the empty implementation to connect the dawgie.pl.farm.

Connect dawgie.pl.farm._res() to dawgie.pl.schedule.purge(). The current behavior is the same.

• purge...

Release 1.2.13

fix security problem (#147)