{ % feed_meta %} Page 5 of 6 for Home | A Few Pinches of Salt

Removing Sensitive Information from Git

There may be a case that you have a git repository that you wish to make public, or distribute to others. In addition to this, there may be sensitive information on this repository that you do not wish to distribute.

The first step, that I believe is the best procedure, if using GitHub for the process, is illustrated in this post.

I will assume that your original repository is private and hosted on GitHub. Next, I will assume that you want to keep that original repository private and work on it into the future. In this case, the best procedure may be to replicate this repository, and make this newly replicated and cleaned a public one (or easily shareable).

The Tools

You will need git and BFG on your system in order for this to work.

The Procedure

The first step is to replicate your original repository.

Cloning

Some of these instructions are taken from here. There are a number of options that this page allows, for example if your copied repository should keep updated with the original one (the procedure that I will be using is the first one).

  1. Make a new private git repository that you are going to be using as the “distributable” version.
  2. Clone a bare copy of that repository:

    $ git clone –bare https://github.com/exampleuser/old-repository.git

  3. Mirror-push to the new repository that you created

    $ cd old-repository.git

    $ git push –mirror https://github.com/exampleuser/new-repository.git

  4. Remove the temporary local repository you created.

    $ cd ..

    $ rm -rf old-repository.git

The next step is to find and replace/remove the sensitive files/information within the repository.

Cleansing

I will be making use of the BFG, which appears to be a better/alternative version to git-filter-branch.

The first step is to download the application, which can be found on the BFG website.

This is a java file, so make sure that you have the Java Runtime Environment installed.

The examples of how to use the program can be found on their website, however, take note that in order to run the application, make use of (take note of the file name that you downloaded):

java -jar bfg.jar

This will be the method on how you will run the bfg command as seen in the examples.

Now that this is out of the way, we can jump into the removal part.

This starts with first cloning the repository that you wish to cleanse:

$ git clone --mirror git://example.com/some-big-repo.git

Now you are ready to start cleaning up.

Passwords and Code

In order to replace text within the codebase, you can make use of a file with a list of the text that you wish to be removed/edited.

The best method to do this is to create a passwords.txt file with a list of the text that you wish to be removed/edited. The following illustrates the different ways this can happen (thanks to this guy, and some info from here):

PASSWORD1                       # Replace literal string 'PASSWORD1' with '***REMOVED***' (default)
PASSWORD2==>examplePass         # replace with 'examplePass' instead
PASSWORD3==>                    # replace with the empty string
regex:password=\w+==>password=  # Replace, using a regex
regex:\r(\n)==>$1 # Replace Windows newlines with Unix newlines

Take note, just like the files, you should have already removed the offending text from the repository, as these pieces of text will also be protected. One can use the command:

egrep -lRZ 'foo' . | xargs -0 -l sed -i -e 's/foo/bar/g'

This replaces ‘foo’ with ‘bar’.

Files

An important note, in the section for files from the BFG site, is that you should first delete your unwanted files and make a commit (and a push) before you run the BFG. The files (in the repo) within the latest commit are protected, which means that the BFG will not be able to delete the unwanted files.

There are cases where you may want to delete an entire folder. This can be done as follows (found here):

$ bfg --delete-folders "{folderA,folderB,folderC}" my-repo.git

Once this is completed, you have to navigate into the git repository.

cd some-repo.git

Then run the following command:

$ git reflog expire --expire=now --all && git gc --prune=now --aggressive

Once this has completed, you are ready to push the changes to this “ready for public” repository.

This completes the process of cleansing your repository.


Setting Up Git

This illustrates the simple steps in setting up git on a windows system.

ssh-keygen -t rsa -b 4096 -C email@email.com

eval $(ssh-agent -s)

ssh-add ~/.ssh/id_rsa

clip < ~/.ssh/id_rsa.pub

This clipped ssh key should then be added to the GitHub SSH keys for the system to work.


Git Make

Running make files on windows is not natively accepted. I wanted to use git bash to do this. In order to get this to work, I found this site, it says how to get the makefile to run.

It also shows you how to get wget, hugo, xpdf, and nano to work.


The First Real Post

The Blog Idea

This is where I talk about why I thought this would be a good idea… Oh how wrong I was/am/are…

I guess that if this blog/post sees the light of day, then I have followed through and that this solution (or one of the following posts) actually worked.

Starting out

There seems to be quite a few posts around the place on how to host your own blog, where the majority of people use… drumroll… Wordpress! I could have gone with the standard Wordpress blog that every Joe and his dog use these days, but decided against it. The second idea I had was to leverage the free services of GitHub with their GitHub Pages service. This system has a few drawbacks… Mostly the main reason I have not chosen to go with this is because it only really supports static site generation, which, can be a good thing, but can be limiting.

Due to a little research over the last few months, I have decided to just jump in and make use of the GitHub pages platform. The first speed bump that I need to fix is the ease of use. I do not like the old school systems that make use of HTML, CSS, and JavaScript files to make websites. Luckily, I found a simple system that allows one to create markdown files that are then converted to blog posts. The use of this Jekyll Now system is how the blog is being run as you see it.


Migrating from Wunderlist Part 2

I just wanted to follow up from my last post on Migrating from Wunderlist.

Now that I have used TickTick for the past couple of months, I have really realised how Wunderlist had been slacking. I guess that is based on the fact that they had not been actively developing for a while 1.

This application that I am using now has, so far, proved to be really reliable and has many more features than I am used to. I don’t use that many to begin with.

  1. Due to the fact that they are in the process of migrating to the Microsoft version of a task tracking system.