Automated Flows For Development#

I do use an IDE for editing code, but for running programs, I still rather use the command-line. This feels way faster to me and it always looks like it gives more options. Also, I like to automate the things I do often, even the small things I need while developing software. When I type in the same commands very often, I find it an opportunity to dust off my bash skills and do something with it.

All these things are a bit hard to share with other team members, because we’re usually using different places for different things (like git repos, scripts, virtual environments, etc). All these scripts can probably improved in many ways in different situations, so I think it also makes it less complex to keep these scripts for yourself and edit those as you please.

Source Shell Scripts#

Because one of the important things I need to do on the command-line is going to the right directory, setting some environment variables and activating other things, I keep a directory called ~/shells which contains scripts that need to be sourced.

A script called ~/shells/project1 might look like this:

#!/bin/bash

export SOME_ENV_VAR=123
cd $HOME/Projects/project1
source venv/bin/activate

Now when I open a terminal and type in . ~/shells/project1, I end up in the right directory with the right environment variables and a python virtual environment activated. The great thing about this:

  • I don’t have to remember all these specific paths and variables

  • I can easily edit those files when things are changing

  • It saves times

Updating Git Projects#

When I work for a project that uses git for version control, I usually work on a feature branch. When my team members merge changes in master, I need to rebase my branch from master to get the latest changes.

Because I don’t want to get too far behind, I execute this script called gupdate every morning.

#!/bin/bash

CURRENT_BRANCH=$(git rev-base --abbrev-ref HEAD)

if [[ "$CURRENT_BRANCH" != "master" ]]; then
  git stash -u
  git checkout master
else
  echo Changes not stashed because current branch is master
fi

git pull
git fetch -p

if [[ "$CURRENT_BRANCH" != "master" ]]; then
  git checkout $CURRENT_BRANCH
  git rebase origin/master
  git stash pop
fi

When I’m working on a feature branch, this script does the following things:

  1. if the current branch is not master, stash all changes (including untracked files) and checkout master.

  2. pull all changes, fetch all other branching and pruning tags

  3. go back to my feature branch

  4. rebase that branch from origin/master

  5. pop changes back from the stash

Updating Libraries#

I usually call the above scripts (sourcing the shell and automatically updating) from another script that is supposed to update everything and shows the latest git status. Such a script is usually called something like project1-gupdate and looks a bit like this:

#!/bin/bash

source $HOME/shells/project1
gupdate
echo <insert more commands here>
git status

The other commands can be related to the following things:

  • updating libraries, like node_modules or python virtual environments

  • executing database scripts for your local database

  • backing or cleaning up things

The git status is to see right away if there are conflicts that need to be resolved.

Testing and Running#

If I have a lot of different project folders with each different ways to run programs and tests, I automate that as well. Usually, I combine these scripts with shell scripts to be sourced, so I can basically run these scripts from every directory. Because the sourcing happens inside the script, you’ll return back to your original directory after the run.

#!/bin/bash

source $HOME/shells/project1
echo <set some environment variables and other configs, maybe even move some files>
echo <insert your run or test commands here>

Final Notes#

  • All my scripts that need to be run are in my ~/bin directory and my shell scripts that need to be sourced are in ~/shells. The ~/bin directory has been added to my $PATH.

  • I try to keep my scripts small and combine them by calling eachother.

  • For related scripts, for example when they’re for the same git project, I use some naming scheme. Usually it’s a prefix with the project name, so I have commands like project1 (for running it), project1-tests, etc.

  • Most projects usually have a Makefile and make commands to automate execution of scripts (locally and in CI pipelines). I would recommend using these commands in your local automation scripts as well.

  • I would also try to make scripts to automatically change branches and set everything (like libraries and databases) straight. If I ever get a chance to do that, I’ll dedicate another blog post to that.

  • I do want to do more with docker and kubernetes locally (at least to keep those skills a bit up to date), but since most of the pipelines are automated and there are not that many issues in the other environments, I haven’t been doing much with that.

  • Know your tools, start small and organize your environment. You can read more about this in one of my old blog posts on Medium here.

Modern Type Hinting in Python   Concurrency with Python