tarantool / luajit / 6936091537 / 1

89%

tarantool/master: 93%

push

github

Fix ABC FOLD rule with constants.

Reported by XmiliaH.

(cherry-picked from commit c8bcf1e5f)

`fold_abc_k()` doesn't patch the first ABC check when the right constant
operand is negative. This leads to out-of-bounds access from the array
on a trace. This patch casts the operands to uint32_t for comparison. If
the right IR contains a negative integer, the second IR will always be
patched. Also, because the ABC check on the trace is unordered, this
guard will always fail.

Also, this fold rule creates new instructions that reference operands
across PHIs. This opens the room for other optimizations (like DCE), so
some guards become eliminated, and we use out-of-bounds access from the
array part of the table on trace. This patch adds the missing
`PHIBARRIER()` check.

Sergey Kaplun:
* added the description and the test for the problem

Part of tarantool/tarantool#9145

5377 of 5986 branches covered (0.0%)

Branch coverage included in aggregate %.

20615 of 23353 relevant lines covered (88.28%)

2747321.4 hits per line