Shrestha/add control dependencies in ngraph cluster

[shresthamalik]

Add control dependency to the translated nGraph nodes. first_ng_op_map stores the entry point nGraph nodes and ng_op_map stores the exit point nGraph nodes for each Tf translated op. After the TF graph has been translated to nGraph, we add the control dependencies to the nGraph nodes using the stored entry and exit points.

[kanvi-nervana]

same as above

[shresthamalik]

ng_filter and ng_input are different data flows, so this check I think is still valid

[sayantan-nervana]

Consider this graph:

X---->Identity----->Z1
         ^    \
         |     \
         Y      Z2

Where X produces the actual data and Y has a control edge on the identity node. What this is saying is that "Do not start computing Z1 and Z2 till Y is done computing".

I think since we have no real op in place of identity, this constraint is probably not translatable right now?

Why might we see this structure? Let us say we have Z1 to Z100 that we want to hold off on computing till Y is done. Instead of attaching control edges from Y to Zi individually, we can attach a control edge to an identity node that feeds all the Zi nodes.

Note however that if we attach the control edge to X, we will conservatively maintain the constraint. Moving the control edge to your data parent is always safe.

Another possibility is moving the control edge to all the Zi nodes

In conclusion: If a TF node has no ng node to which we can attach incoming control edges, then we can either attach the control edge to the input ngraph node (which respects the intended constraints, but introduces additional constraints), or we can attach the control edge to the next nodes (this is the correct translation of the constraint, however it might need more bridge work, since we have to keep track of unattached control edges, which must be taken care of by the children/grandchildren nodes. (consider the scenario of X->identity->identity->Z, with a control edge incoming into the first identity. This graph structure is unlikely, but valid) )

[mingshan-wang]

LGTM! Thanks for making all the changes.

[sindhu-nervana]

LGTM, thanks for answering all questions.

[sayantan-nervana]

In our discussion @shresthamalik pointed out this graph:

    W
    |
    v
X-->Y-->*I--->Z
|        ^
|________|

Data edges: X->Y, I->Z, X->I, W->Y. Control edges (starred): Y->*I. I is identity node

If we attach the control edge Y->I to Y->X (parent of I), then we introduce a cycle in nodes X,Y

In such cases we cannot attach control edges to parents, and . we cannot honor the Y->*I control edge (unless we attach the control edge to children Z)