Valloric / ycmd / 1857

94%

push

travis-ci

Auto merge of #455 - puremourning:unicode-investigation, r=Valloric

[READY] Fix issues with multi-byte characters

## Summary

This change introduces more general support for non-ASCII characters in buffers handled by YCMD.

In ycmd's public API, all offsets are byte offsets into the UTF-8 encoded buffers. We also assume (because, we have no other choice) that files stored on disk are also UTF-8 encoded. Internally, almost all of ycmd's functionality operates on unicode strings (python 2 `unicode()` and python 3 `str()` objects, transparently via `future`). Many of the downstream completion engines expect unicode code points as the offsets in their APIs. One special case is the `ycm_core` library (identifier completer and clang completer), which requires instances of the _native_ `str` type. All strings used within the c++ using `boost::python` require passing through `ToCppStringCompatible`

Previously, we were largely just assuming that `code point == byte offset` - i.e. all buffers contained only ASCII characters. This worked up to a point, but more by luck than judgement in a number of places.

## References

In combination with a YCM change and PR https://github.com/Valloric/ycmd/pull/453, I hope this:

- fixes https://github.com/Valloric/ycmd/issues/109
- fixes https://github.com/Valloric/YouCompleteMe/issues/2096
- fixes https://github.com/Valloric/YouCompleteMe/issues/2088
- fixes https://github.com/Valloric/YouCompleteMe/issues/2069
- fixes https://github.com/Valloric/YouCompleteMe/issues/2066
- fixes https://github.com/Valloric/YouCompleteMe/issues/1378

## Overview of changes

The changes fall into the following areas:

- Providing access to and conversion to/from code points and byte offsets (`request_wrap.py`)
- Changing certain algorithms/features to work entirely in codepoint space when they are trying to operate on logical 'characters' within the buffer (see known issues for why this isn't perfect, but probably most of the way there)
... (continued)

125 of 125 new or added lines in 14 files covered. (100.0%)

3028 of 3574 relevant lines covered (84.72%)

0.85 hits per line