graph, example, benchdnn: add int4 gated mlp #2252
Conversation
TaoLv
commented
Dec 12, 2024
TaoLv
commented
- Define Gated MLP patterns with int4 compressed weights.
- Add the example and benchdnn test file accordingly.
| pm::pb_op_t *deq_up = pgraph->append_op( | ||
| graph::op_kind::DynamicDequantize); | ||
| pm::pb_op_t *deq_gate = pgraph->append_op( | ||
| graph::op_kind::DynamicDequantize); |
There was a problem hiding this comment.
Should those be restricted to that same in_edge?
There was a problem hiding this comment.
Hi Mourad, the dequantize ops here are for weights which are different for the two matmul ops (up and gate). But I guess you meant the src should be same for the two matmul ops. Currently, the pattern and the primitive based implementation support different src inputs for the two matmul ops. Do you think we'd better add a restriction for that?
There was a problem hiding this comment.
Thanks Tao. Yes I meant the src. IIRC, the gated MLP pattern always have shared src, correct?
I guess the question is hence the actual pattern definition: if optimizations can leverage that src is shared by both matmul (e.g. by keeping reordered buffer in SLM on GPU), it would be safer to define the pattern as such IMHO (and reflect that in tests as well).
|
make test |
