Skip to content

Event Flow of a Spark setup

fengjingchao edited this page Nov 27, 2014 · 2 revisions

This page describe a typical event flow of a spark like setup. Since each tree can be decomposed into multiple path, each path connect the root to a leaf node, so it is sufficient to describe the event flow along the path. To make it simple we describe the event flow on a three nodes chain (or path): A -> B -> C, where A is parent of B and B is parent of C.

In a typical epoch:
A FlagParentMetaReady
B got notified with ParentMetaReady
B invoke framework.DataRequest from A to get parent data (parameter in spark settings)
B got notified with ParentDataReady, it then start the local computation, and FlagParentMetaReady.
C got notified with ParentMetaReady
C invoke framework.DataRequest from B to get parent data (again parameter)
C got notified with ParentDataReady, it finished local computation, and FlagChildMetaReady (since it is leaf node)
B got notified with ChildMetaReady (from C)
B invoke framework.DataRequest from B to get child data (gradient in spark setting)
B got notified with ChildDataReady, B merges child data with local copy and FlagChildMetaReady
A got notified with ChildMetaReady
A invoke framework.DataRequest form B to get child data
A got notified with ChildDataReady, merges child data from all children, and decide whether there should be another epoch, if so, it invoke framework.IncEpoch.