cantino / huginn

89%

master: 92%

LAST BUILD ON BRANCH web_content_charset

branch: web_content_charset

CHANGE BRANCH

x

Reset

• web_content_charset
• Alex-Ikanow-website_deduplication_improvement
• Warloxx-rss_agent_dynamic_cleanup
• ZirconCode-master
• add_agent_age_column
• add_headers_to_website_agent
• add_keep_events_for_to_agents
• add_media_namespace
• add_optional_jsonpath_eval
• add_unescape_filter
• agent_editing_returns
• agents_in_gems
• alias1-196-disable-agents
• allow-style-attributes
• allow_any_agent_to_me_commanded
• allow_from_in_email_agents
• allow_liquid_in_email_recipients
• allow_url_from_event_to_be_array
• bind_all_interfaces
• bootstrap3-tweeks
• bump_gems
• bump_weibo_2
• clarify_delayed_job_web_usage
• clear-memory
• commander_agent_can_configure
• cve-2015-7499
• data_output_agent
• data_output_agent_can_build_complex_xml
• data_output_agent_cannot_create_events
• data_output_agent_limits_events_after_ordering
• data_output_agent_push
• db_less_strict
• delay_agent
• diagram_layout
• diagram_toggle_disabled
• do_not_raise_on_missing_credential
• downgrade_better_errors
• dry_run_from_anywhere
• dry_run_with_event
• dryrun_event_formatting
• dsander-omniauth
• email_agent_body
• email_in_development_option
• enable_github_integration
• encoding_detection
• enhance-ar_mysql_column_charset
• enhance-data_output_agent
• enhance_agent_picker
• event_does_not_require_url
• event_update_should_not_touch_last_event_at
• explain_propagate_immediately
• expose_agent_id
• extend_change_detector_agent
• fix-invalid-json-save
• fix_1940
• fix_account_editing
• fix_capistrano_db_sync
• fix_credential_create_editor
• fix_decoding_in_rss_agent
• fix_header_prop_link
• fix_heroku_instructions
• fix_http_status_agent
• fix_regex_replace
• fix_regexp_extraction
• fix_rss_agent_ids
• fix_sass_warnings
• fix_scenarios_pagination_ui
• fix_schedule_naming
• fix_schedule_reset
• fix_seed
• fix_translate_agent
• fix_website_agent_url_handling
• gap_detector_agent
• gem_groups
• github_agent
• google_cal_agent_accepts_liquid
• guess_rss_encoding
• gzip_support
• handle_dups
• handle_removed_events
• heroku
• heroku_first_time_makes_admin_user
• human_task_agent_separate_events
• ignore-fake-key
• ignore_duplicate_keywords
• ignore_empty_author
• imap_agent_scrub_strings
• improve_uri_normalization
• include_sort_info
• increase_default_database_pool
• interpolate_headers
• interpolated_options
• javascript-agent
• job_polling_fix
• js_agent_can_be_dryrun
• jsagent_fix_memory
• keep_events_for_in_seconds
• link_to_huginn_agent
• link_to_liquid
• liquid_regex_replace_block
• longer_agent_logs
• maintain_changes
• manual_event_agent_supports_liquid
• many_gems_are_optional
• master
• minor_admin_cleanup
• minor_improvements
• mock_web_request
• more-nitrous-removals
• more_robust_feature_specs
• move_back_to_current_dj
• multiple_email_recipients
• nullify_agent_log_event_refs
• object_logging
• odds_and_ends
• omniauth_refactors
• only_show_headline_if_present
• openshift
• openshift2
• optionally_load_tumblr_gem
• output_agent_can_order
• post_agent_additions
• post_agent_downcase_response_headers
• post_agent_emits_events
• post_agent_headers
• post_agent_interpolates_url
• prevent_event_replay_on_linking
• prevent_extra_threads_in_specs
• pushover_parameters
• raise_pool_default_size
• reduce_default_delayed_job_max_runtime
• refactor_twitter_concern
• remove-nitrous
• remove-openshift
• remove-unused-contact-model
• remove_beeper_agent
• remove_spring
• remove_unused_env_vars
• rescue_json_errors_on_deserialize
• rescue_mail_errors
• retire_ar_mysql_column_charset
• revert-1071
• revert-1974-fix_1940
• rss_agent
• rss_agent-events_order
• rss_agent-feedjira
• rss_agent-rss_parser
• rss_agent_can_take_multiple_urls
• rss_agent_event_order
• rss_agent_logs_exact_feed_on_error
• rss_agent_podcast_tags
• rss_agent_sort_info
• rss_agent_sorts_all_events
• rss_content_type
• scenarios
• scenarios_contain_control_links
• scenarios_contain_control_links_rebased
• security_bumps
• semicolon_passwords
• set_default_port
• set_global_email_settings
• shell_command_agent_enhancement
• shell_out_unbundled
• shorter_twitter_reload
• simplify_log_format
• site-icons
• smtp_ssl_verification_skip_option
• stvnrlly-master
• support_ruby2.4
• table_sorting
• timestamps_in_twitter_agent
• travis_mysql
• trigger_agent_can_reemit_event
• trigger_agent_supports_partial_matches
• truncate_agent_log_messages
• try_updating_twitter
• twitter_is_optional_too
• update-nodejs-for-chef
• update-nokogiri
• update_changes_file
• update_gemfile_for_libv8
• update_gems_for_security
• update_nokogiri_again
• update_rails
• update_rails_and_gems
• update_rails_to_4_2_5_1
• update_readme
• upgrade-forecast-io-gem
• upgrade-liquid
• upgrade_dependencies
• upgrade_liquid_to_3
• upgrade_rspec
• upgraded_post_agent
• uri_expand
• use_boolify_for_disable_ssl_verification
• user_credentials
• we_do_not_have_a_github_agent_yet
• webhook_agent_can_iterate
• webhook_recaptcha
• website_agent-url_on_receive
• website_agent_can_interpolate_after_extraction
• website_agent_can_parse_payload
• website_agent_jsonpath_validation
• website_agent_loop_variable
• website_agent_normalize_uri
• website_agent_repeat_option
• website_agent_resolves_url_optionally
• website_agent_template_merge
• xpath-string

Implement charset handling in WebRequestConcern

- The `force_encoding` and `unzip` options in WebsiteAgent are moved to
  WebRequestConcern so other users of the concern such as RssAgent can
  benefit from them.

- WebRequestConcern detects a charset specified in the Content-Type
  header to decode the content properly, and if it is missing the
  content is assumed to be encoded in UTF-8 unless it has a binary MIME
  type.  Not all Faraday adopters handle character encodings, and
  Faraday passes through what is returned from the backend, so we need
  to do this on our own. (cf. lostisland/faraday#139)

- WebRequestConcern now converts text contents to UTF-8, so agents can
  handle non-UTF-8 data without having to deal with encodings
  themselves.  Previously, WebsiteAgent in "json"/"text" modes and
  RssAgent would suffer from encoding errors when dealing with non-UTF-8
  contents.  WebsiteAgent in "html"/"xml" modes did not have this
  problem because Nokogiri would always return results in UTF-8
  independent of the input encoding.

This should fix #608.