stanford-oval / genie-toolkit / 1634 / 2

54%

master: 56%

push

travis-ci-com

Refactor how entities are generated in agent sentences (#453)

The goal of this refactor is two fold:
- We need to generate delexicalized agent utterances at inference
  time to pass to the NLG
- We want the actual text of the entity to be generated by the I18n
  code during postprocessing, so consistent postprocessing is
  applied to both synthetic and NLG sentences, and we can use
  locale-specific code.

To apply this refactoring, the system of inference time constants
is modified to make use of a SequentialEntityAllocator, which allocates
entity tokens (NUMBER_0, DATE_1, etc.) for each value that appears
in the context. This entity allocator is also passed to the Describe
module when that module extracts a value from the program.

At training time, we have only one SequentialEntityAllocator,
which is primed with the fake entities used during generation,
plus any entity that is generated by the simulator and observed
by Describe. At inference time, we reset the entity allocator
immediately prior to processing a new context.

The entities thus allocated are later passed to the I18n module for
postprocessing. This module gained code to automatically select
the most appropriate unit for a given size.

Fixes #447

3726 of 15250 branches covered (24.43%)

Branch coverage included in aggregate %.

8945 of 28005 relevant lines covered (31.94%)

36434.11 hits per line