airbnb / airflow / 2764
0%

DEFAULT BRANCH: master
Ran
Jobs 6
Files 125
Run time 3min
Badge
Embed ▾
README BADGES
x

If you need to use a raster PNG badge, change the '.svg' to '.png' in the link

Markdown

Textile

RDoc

HTML

Rst
2764

push

travis-ci

Chris Riccomini 
[AIRFLOW-179] DbApiHook string serialization fails when string contains non-ASCII characters

Dear Airflow Maintainers,

Please accept this PR that addresses the following issues:
- https://issues.apache.org/jira/browse/AIRFLOW-179

In addition to correctly serializing non-ASCII characters the literal transformation also corrects an issue with escaping single quotes (').

Note it was my intention to add another unit test to `test_hive_to_mysql` in `tests/core.py` however on inspection the indentations of the various methods seemed wrong, methods are nested and it's not apparent what class they refer to. Additionally it seems a number of the test cases aren't related to the corresponding class.

For testing purposes I simply ran a pipeline which previously failed with the following exception,

    [2016-05-26 22:03:39,256] {models.py:1286} ERROR - 'ascii' codec can't decode byte 0xc3 in position 230: ordinal not in range(128)
    Traceback (most recent call last):
      File "/usr/local/lib/python2.7/dist-packages/airflow/models.py", line 1245, in run
    result = task_copy.execute(context=context)
      File "/usr/local/lib/python2.7/dist-packages/airflow/operators/hive_to_mysql.py", line 88, in execute
    mysql.insert_rows(table=self.mysql_table, rows=results)
      File "/usr/local/lib/python2.7/dist-packages/airflow/hooks/dbapi_hook.py", line 176, in insert_rows
    l.append(self._serialize_cell(cell))
      File "/usr/local/lib/python2.7/dist-packages/airflow/hooks/dbapi_hook.py", line 196, in _serialize_cell
    return "'" + str(cell).replace("'", "''") + "'"
      File "/usr/local/lib/python2.7/dist-packages/future/types/newstr.py", line 102, in __new__
    return super(newstr, cls).__new__(cls, value)
    UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 230: ordinal not in range(128)

and verified with the presence of the fix that the task succeeded and the resulting output was correct. Note currently from grokking the code base it seems that only `MySqlHook` objects call the the `insert_rows` method.

Author: John Bodley <john.bodley@airbnb.com>

Closes #1550 from johnbodley/dbapi_hook_serialization.

5501 of 8225 relevant lines covered (66.88%)

3.99 hits per line

Jobs
ID Job ID Ran Files Coverage
1 2764.1 (TOX_ENV=py27-cdh-airflow_backend_mysql) 0
66.59
 Travis Job 2764.1
2 2764.2 (TOX_ENV=py27-cdh-airflow_backend_sqlite) 0
66.7
 Travis Job 2764.2
3 2764.3 (TOX_ENV=py27-cdh-airflow_backend_postgres) 0
66.59
 Travis Job 2764.3
4 2764.4 (TOX_ENV=py34-cdh-airflow_backend_mysql) 0
66.32
 Travis Job 2764.4
5 2764.5 (TOX_ENV=py34-cdh-airflow_backend_sqlite) 0
66.41
 Travis Job 2764.5
6 2764.6 (TOX_ENV=py34-cdh-airflow_backend_postgres) 0
66.33
 Travis Job 2764.6
Source Files on build 2764
Detailed source file information is not available for this build.