tarantool / luajit / 12808148437

93%

tarantool/master: 93%

push

github

x86/x64: Add more red zone checks to assembler backend.

Thanks to Peter Cawley.

(cherry picked from commit d854d00ce)

Assembling some instructions (like `IR_CONV int.num`, for example) with
many mcode to be emitted may overflow the `MCLIM_REDZONE` (64) at once
due to the huge mcode emitting.

For example `IR_CONV` in this test requires 66 bytes of the
machine code:
|  cvttsd2si r15d, xmm5
|  xorps xmm9, xmm9
|  cvtsi2sd xmm9, r15d
|  ucomisd xmm5, xmm9
|  jnz 0x11edb00e5       ->37
|  jpe 0x11edb00e5       ->37
|  mov [rsp+0x80], r15d
|  mov r15, [rsp+0xe8]
|  movsd xmm9, [rsp+0xe0]
|  movsd xmm5, [rsp+0xd8]

The reproducer needs sufficient register pressure as to immediately
spill the result of the instruction to the stack and then reload the
three registers used by the instruction, and to have chosen enough
registers with numbers >=8 (because shaving off a REX prefix [1] or two
would get 66 back down to <= `MCLIM_REDZONE`), and to be using lots of
spill slots (because memory offsets <= 0x7f are shorter to encode
compared to those >= 0x80. So, each reload instruction consumes 9 bytes.
This makes this reproducer unstable (regarding the register allocator
changes). Thus, only original test case is added as a regression test.

This patch adds the red zone overflow checks more often for the IRs with
many instructions to be emitted.

Sergey Kaplun:
* added the description for the problem

[1]: https://wiki.osdev.org/X86-64_Instruction_Encoding#REX_prefix

Part of tarantool/tarantool#10709

5696 of 6034 branches covered (94.4%)

Branch coverage included in aggregate %.

4 of 4 new or added lines in 1 file covered. (100.0%)

21696 of 23431 relevant lines covered (92.6%)

2935711.62 hits per line