Creating custom yellow handshake emojis with zero-width joiners

Is there any specification on how "far" a zero-width joiner can reach? I thought a ZWJ would only affect the codepoints directly adjacent to it, but here we have sequence of 5 codepoints and only one ZWJ for all of them (which is also the only codepoint that signals that "special treatment" should be applied here).

So how would a parser know all five codepoints make up a grapheme cluster and not just the inner three?

Edit: Ok, didn't realize the "Fitzpatrick" codepoints are modifiers that I guess always refer to the previous codepoint. So there is essentially an "operator precedence" defined between modifiers and ZWJs.