splitgraph / splitgraph / 1403 / 1

96%

master: 94%

push

travis-ci-com

Speed up chunking large tables by about 4x.

Instead of using a ton of LIMIT queries and sorting the table every time \+ copying every object into a staging area to add the updated/deleted flag, precompute partitions once in a temporary table (still uses up space but is required because we can't quickly add the same rank column to the original table) by using `ROW_NUMBER` and then copy data over into the actual CStore object directly.

Also use temporary tables in a couple more spaces and pass the table schema in instead of redundantly recomputing it.

Finally, remove sorting when computing LtHash (it works by defining an algebraic addition operation on hashes, so order doesn't matter).

This means that a dataset with 8M rows now can actually get chunked and hashed in ~6 minutes as opposed to ~24 minutes that it took previously.

As a side effect of passing the table schema anywhere, objects seem to store their original ordinals (e.g. deleting a column from the middle of a table doesn't change the numbering of any other columns) -- not sure if this is a good side effect; also noticed that ordinals are included in schema hashes and that the update-delete flag is at the beginning of the table for patch fragments and at the end of it for base fragments (which also affects hashing).

2164 of 2322 branches covered (93.2%)

Branch coverage included in aggregate %.

4763 of 4956 relevant lines covered (96.11%)

0.96 hits per line