14431780412

Committed 13 Apr 2025 05:35PM UTC coverage: 88.616% (+5.5%) from 83.114%

Build # 14431780412

Build Type

push

github

Committed by

web-flow

Commit Message

Ingestion v2.0.0 (#94)

### Description
Feature Flags and File Directory Modernization

## Added
- Singleton Feature Flag Class + Decorator
- The Class allows the Feature Flags to be loaded from the Database once
at script runtime and then used in every scraper function via a
Decorator

## Updated
- Re-arranged `src/` File Directory structure to split the
`src/utils.py` file out into individual files based on function type
- Feature Flags Functionality to use a Decorator to check whether
functions need to be ran, rather than including `feature_flags_df` in
every function zzz
- Renamed `record_time` Decorator to `record_function_time_decorator`
- `README.md` to include link to [doqs](https://doqs.jyablonski.dev) as
well as to remove comment about credentials

## Deleted
- Sentry SDK Package
- Outdated Code that was commented out; I left this inside the
`src/logs/*` directory locally

Run Details

526 of 619 new or added lines in 13 files covered. (84.98%)

2 existing lines in 1 file now uncovered.

903 of 1019 relevant lines covered (88.62%)

0.89 hits per line

Source File
Press 'n' to go to next uncovered line, 'b' for previous

66.67

/src/database.py

import logging

import pandas as pd


def write_to_sql(con, table_name: str, df: pd.DataFrame, table_type: str) -> None:
    """
    Simple Wrapper Function to write a Pandas DataFrame to SQL

    Args:
        con (SQL Connection): The connection to the SQL DB.

        table_name (str): The Table name to write to SQL as.

        df (DataFrame): The Pandas DataFrame to store in SQL

        table_type (str): Whether the table should replace or append to an
            existing SQL Table under that name

    Returns:
        Writes the Pandas DataFrame to a Table in the Schema we connected to.

    """
    try:
        if len(df) == 0:
            logging.info(f"{table_name} is empty, not writing to SQL")
        else:
            df.to_sql(
                con=con,
                name=table_name,
                index=False,
                if_exists=table_type,
            )
            logging.info(
                f"Writing {len(df)} {table_name} rows to aws_{table_name}_source to SQL"
            )

        return None
    except Exception as error:
        logging.error(f"SQL Write Script Failed, {error}")
        return None

1	import logging	1✔
2
3	import pandas as pd	1✔
4
5
6	def write_to_sql(con, table_name: str, df: pd.DataFrame, table_type: str) -> None:	1✔
7	"""
8	Simple Wrapper Function to write a Pandas DataFrame to SQL
9
10	Args:
11	con (SQL Connection): The connection to the SQL DB.
12
13	table_name (str): The Table name to write to SQL as.
14
15	df (DataFrame): The Pandas DataFrame to store in SQL
16
17	table_type (str): Whether the table should replace or append to an
18	existing SQL Table under that name
19
20	Returns:
21	Writes the Pandas DataFrame to a Table in the Schema we connected to.
22
23	"""
24	try:	1✔
25	if len(df) == 0:	1✔
NEW 26	logging.info(f"{table_name} is empty, not writing to SQL")	×
27	else:
28	df.to_sql(	1✔
29	con=con,
30	name=table_name,
31	index=False,
32	if_exists=table_type,
33	)
34	logging.info(	1✔
35	f"Writing {len(df)} {table_name} rows to aws_{table_name}_source to SQL"
36	)
37
38	return None	1✔
NEW 39	except Exception as error:	×
NEW 40	logging.error(f"SQL Write Script Failed, {error}")	×
NEW 41	return None	×

jyablonski / python_docker / 14431780412

Source File Press 'n' to go to next uncovered line, 'b' for previous

Source File
Press 'n' to go to next uncovered line, 'b' for previous