splitgraph / splitgraph / 1394 / 1

96%

master: 94%

push

travis-ci-com

Add max API request length and max amount of variadic arguments that an API query can have. Add chunking/adaptive batching to API calls that can reach the max request length.

* Add `run_chunked_sql` that works on API functions that take arrays (splitting them up into multiple calls with arrays of limited size): `get_object_meta`, `get_object_upload_urls` etc
* Change `run_sql_batch` to fill the batch up to the API request size limit instead of default 1000 arguments: `register_object`
* For `register_tables`, it's more complicated as we want to be able to batch up calls registering many small tables but also not breach the API request size limit if there's a table with a few hundred objects.
  * Also changed `add_table` SQL API call allowing it to be called multiple times which will add extra objects to an existing table -- this is necessary since we might not be able to register a table in one call.
* Set a WIP default for max query size at 32KiB (subject to change). Some sample numbers:
  * `add_object` call registering 20 objects from the US election dataset with range index but without bloom index is 31.6KiB (about 1.6KiB per object)
  * With a bloom index on `county_fips` this becomes 1.9KiB per object.
  * With a bloom index on `id` (pathological case: there are 100000 rows per object and the same amount of distinct IDs) this becomes 160KiB of metadata per object.
    * **Note this means this object can't be registered with the current settings**.
    * Arguably, it's OK to enforce such a limit: if an index is this large, it might be useless (like in this case) or we should store it externally somehow.
    * Also, this maps to a single Postgres record which will get TOASTed if it's above 2KB, so we might want to limit the size of a record anyway.
  * A single `add_table` call can fit about 450 objects before having to be split up (probably also fine: with our default object creation settings that's pointing at a 4.5M row table)

2149 of 2296 branches covered (93.6%)

Branch coverage included in aggregate %.

4753 of 4937 relevant lines covered (96.27%)

0.96 hits per line