Design Ideas

Topic: Workflows

Problem: Celery flows can be cumbersome to monitor & debug

Proposal: Replace with a DAG workflow systems like:

• DoIt
• Apache Airflow
• Luigi

More reading:

• http://nerds.airbnb.com/airflow/
• https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Links
• Airflow Can handle really large files like instance images using Fileflow

Topic: Error handling and reporting

Problem: Balancing friendly user errors with easy triaging and security

Proposal: Ticketing system like web2py has

• Show a friendly message and a ticket number to the user
• Save a full stack trace with the same ticket number so an admin can use it when triaging problems

Links:

• http://web2py.com/books/default/chapter/29/04/the-core (search for 'ticket')
• https://djangopackages.org/grids/g/ticketing/