Scripting for better devops: a Python kata
I like automating things, so I try cutting edges where possible. One of those tiny annoying daily routines was figuring the correct cluster to select with kubectl, when only a country name is known. Surely, if someone has just 2-3 countries, or even regional deployment, this is not a problem at all. On the contrary, if the deployment has 30, 40, 70 countries, then it becomes a challenge.
Another common task would be to proxy the database connection to a country database, live or staging.
Lastly, automating the process of exporting this database and subsequently importing it into my local database would be a good candidate for automation.
So, I've opted to create a script.
Initially, I considered writing a script using NodeJS, however, not every engineer has the NodeJS runtime at their disposal. This poses significant distribution challenges, particularly in case if the script is to be integrated into a project repository. Bash scripting was also off the table. Despite it's ubiquity, I find this language quite unwieldy, somehow cumbersome and not well suited for scripts of substantial complexity.
Eventually I've decided to give Python a try. After all, Python is a language of many applications: system scripting, plugin development, cloud native development.
#!/usr/bin/env python3# -*- coding: utf-8 -*-import argparseimport subprocessfrom typing import ListLIVE_PROJECT_NAME = "my-project-live"STG_PROJECT_NAME = "my-project-stg"SRC_DB_NAME = "project-db"SRC_DB_USER_NAME = "project-db-username"DST_DB_NAME = "project-db"DST_DB_USER_NAME = "project-db-username"DST_DB_PASSWORD = ""DST_DB_PORT = "5432"DUMP_FILE_NAME = "/tmp/database-dump.sql"DUMP_ALTERED_FILE_NAME = "/tmp/database-dump-altered.sql"regions = {"eu": "europe-west1","us": "us-east1",}clusters = {"eu": "gke_my-project-live_europe-west1_01","us": "gke_my-project-live_us-east1_01",}countries_live: List[Country] = [Country(name="USA", code="us", region="us", db_instance="live-us"),Country(name="Germany", code="de", region="eu", db_instance="live-de"),]countries_stg: List[Country] = [Country(name="USA", code="us", region="us", db_instance="stg-us", project=STG_PROJECT_NAME),]countries_local: List[Country] = [Country(name="Local", code="lo", region="lo", db_instance="local", project=""),]class Country:name: str = ""code: str = ""region: str = ""db_instance: str = ""db_instance_replica: str = ""project_name: str = LIVE_PROJECT_NAMEdef __init__(self, name: str = "", code: str = "", region: str = "", db_instance: str = "", db_instance_replica: str = "", canary: bool = False, project_name: str = LIVE_PROJECT_NAME):self.name = nameself.code = codeself.region = regionself.db_instance = db_instanceself.db_instance_replica = db_instance_replicaif self.db_instance_replica == "":self.db_instance_replica = self.db_instance+"-replica"self.project_name = project_namepassdef get_connection(self) -> str:return "{}:{}:{}".format(self.project_name, regions[self.region], self.db_instance)def get_replica_connection(self) -> str:return "{}:{}:{}".format(self.project_name, regions[self.region], self.db_instance_replica)def get_name_upper(self) -> str:return self.name.upper()def get_code_upper(self) -> str:return self.code.upper()def main():args = parse_arguments()action = args.actionenv = args.envcountry = args.countryrw = args.rwif action == "proxy_database":found_country = find_country(country, env)if found_country == None:print("No such country: {}".format(country))exit(1)if rw:print("Caution: read-write mode enabled!")proxy_database(found_country, rw)elif action == "select_cluster":found_country = find_country(country, env)if found_country == None:print("No such country: {}".format(country))exit(1)select_cluster(found_country)elif action == "dump_database":dump_database()exit(0)def select_cluster(country: Country):kubectl_cmd = get_absolute_path("kubectl")execute("{} config use-context {}".format(kubectl_cmd, clusters[country.region]))def proxy_database(country: Country, rw: bool):connection = country.get_connection()if rw:connection = country.get_replica_connection()cloud_sql_proxy_cmd = get_absolute_path("cloud_sql_proxy")execute("{} -instances={}=tcp:54321".format(cloud_sql_proxy_cmd, connection))def dump_database() -> None:print("This action dumps the remote database to the local Postgres instance. Make sure the SQL proxy is running.")src_db_password = getpass.getpass("Source database password: ")if src_db_password == "":print("The password may not be empty")exit(1)src_connection_string = "postgres://{}:{}@127.0.0.1:54321/{}".format(SRC_DB_USER_NAME, src_db_password, SRC_DB_NAME)dst_connection_string = "postgres://{}:{}@127.0.0.1:{}".format(DST_DB_USER_NAME, DST_DB_PASSWORD, DST_DB_PORT)pg_connection_string = "{}/postgres".format(dst_connection_string)print("Running this action will overwrite your local database. Type \"yes\" to proceed.")user_confirmation = input()if user_confirmation != "yes":print("Operation cancelled")exit(0)print("Dumping the remote database...")execute_no_echo("pg_dump {} > {}".format(src_connection_string, DUMP_FILE_NAME))execute("sed -e \"s/OWNER TO {}/OWNER TO {}/g\" {} > {}".format(SRC_DB_USER_NAME, DST_DB_USER_NAME, DUMP_FILE_NAME, DUMP_ALTERED_FILE_NAME))execute("psql \"{}\" -c \"drop database if exists \\\"{}\\\"\"".format(pg_connection_string, DST_DB_NAME))execute("psql \"{}\" template1 -c \"create database \\\"{}\\\" with owner {}\"".format(pg_connection_string, DST_DB_NAME, DST_DB_USER_NAME))print("Importing the database locally...")execute("psql \"{}/{}\" < {}".format(dst_connection_string, DST_DB_NAME, DUMP_ALTERED_FILE_NAME))execute("rm {}".format(DUMP_FILE_NAME))execute("rm {}".format(DUMP_ALTERED_FILE_NAME))print("Done!")def execute(command):print("Executing: {}".format(command))try:subprocess.run(command, shell=False)except KeyboardInterrupt:print("Ctrl+C pressed. Exiting gracefully.")def execute_no_echo(command):try:subprocess.run(command, shell=False)except KeyboardInterrupt:print("Ctrl+C pressed. Exiting gracefully.")def get_absolute_path(cmd):output = subprocess.run("which {}".format(cmd), capture_output=True, shell=True)return output.stdout.decode()[:-1]def find_country(country: str, env: str) -> Country:country_list = countries_stgif env = "live":country_list = countries_liveelif env = "lo":country_list = countries_localcountry_uppercased = country.upper()for country_element in countries:if (country_element.get_name_upper() == country_uppercased or country_element.get_code_upper() == country_uppercased):return country_elementreturn Nonedef parse_arguments():parser = argparse.ArgumentParser(description="A helper tool for faster devops")parser.add_argument("country", type=str, help="The country name or code")parser.add_argument("env", choices=["live", "stg", "lo"], help="Environment")parser.add_argument("action", choices=["proxy_database", "select_cluster", "dump_database"], help="What to do")parser.add_argument("--rw", action="store_true", help="If set to TRUE, the connection will be writable")return parser.parse_args()if __name__ == "__main__":main()
It is recommended to run a python script within a dedicated virtual environment. To do so, type:
python -m venv .venvsource ./.venv/bin/activate
I don't have any external dependencies, but should I have any, the following command must also be executed:
pip install -r requirements.txt
When running the script the action must be specified, along with the other parameters:
python ./devops_swissknife.py --action proxy_database --env live --country de --rw
Well, this script has proven to be useful, I will be adding new features if I see fit.
Sergei Gannochenko
Golang, React, TypeScript, Docker, AWS, Jamstack.
19+ years in dev.