1234

pending completion

Build # 1234

Build Type

push

travis-ci

Committed by

web-flow

Commit Message

Fast segmentation (#229)

Improves text segmentation engine by switching from a regex-based approach to a state table-based approach.

Run Details

244 of 244 new or added lines in 16 files covered. (100.0%)

126423 of 132280 relevant lines covered (95.57%)

259254.22 hits per line

Source File
Press 'n' to go to next uncovered line, 'b' for previous

85.19

/lib/twitter_cldr/segmentation/category_table.rb

# encoding: UTF-8

# Copyright 2012 Twitter, Inc
# http://www.apache.org/licenses/LICENSE-2.0

require 'base64'

module TwitterCldr
  module Segmentation
    class CategoryTable
      PACK_FMT_16 = 'NNn'.freeze

      class << self
        def load16(data)
          data = Base64.decode64(data)

          new(
            (0...data.size).step(10).map do |i|
              data[i...(i + 10)].unpack(PACK_FMT_16)
            end
          )
        end
      end

      attr_reader :values

      def initialize(values)
        @values = values
      end

      def get(codepoint)
        find(codepoint)[2]
      end

      def dump16
        data = ''.b.tap do |result|
          values.each do |vals|
            result << vals.pack(PACK_FMT_16)
          end
        end

        Base64.encode64(data)
      end

      private

      def find(codepoint)
        values.bsearch do |entry|
          next -1 if codepoint < entry[0]
          next 1 if codepoint > entry[1]
          0
        end
      end
    end
  end
end

1	# encoding: UTF-8
2
3	# Copyright 2012 Twitter, Inc
4	# http://www.apache.org/licenses/LICENSE-2.0
5
6	require 'base64'	7✔
7
8	module TwitterCldr	7✔
9	module Segmentation	7✔
10	class CategoryTable	7✔
11	PACK_FMT_16 = 'NNn'.freeze	7✔
12
13	class << self	7✔
14	def load16(data)	7✔
15	data = Base64.decode64(data)	28✔
16
17	new(	28✔
18	(0...data.size).step(10).map do \|i\|	28✔
19	data[i...(i + 10)].unpack(PACK_FMT_16)	65,366✔
20	end
21	)
22	end
23	end
24
25	attr_reader :values	7✔
26
27	def initialize(values)	7✔
28	@values = values	28✔
29	end
30
31	def get(codepoint)	7✔
32	find(codepoint)[2]	292,236✔
33	end
34
35	def dump16	7✔
36	data = ''.b.tap do \|result\|	×
37	values.each do \|vals\|	×
38	result << vals.pack(PACK_FMT_16)	×
39	end
40	end
41
42	Base64.encode64(data)	×
43	end
44
45	private	7✔
46
47	def find(codepoint)	7✔
48	values.bsearch do \|entry\|	292,236✔
49	next -1 if codepoint < entry[0]	3,004,820✔
50	next 1 if codepoint > entry[1]	948,801✔
51	0	292,236✔
52	end
53	end
54	end
55	end
56	end

twitter / twitter-cldr-rb / 1234

Source File Press 'n' to go to next uncovered line, 'b' for previous

Source File
Press 'n' to go to next uncovered line, 'b' for previous