{ % feed_meta %} Page 3 of 4 for Home | A Few Pinches of Salt

Allowing Tags

I have, for a while, known that tags are often an important part of any website. I did not know exactly how to implement these tags within the github pages system. A little searching and I have found this method which appears to be the best approach as it still makes use of github pages and Jekyll.

I have currently updated all of my posts so that they include a number of tags. I will soon attempt to update the site so that it can make use of these tags.


Sankey Generation

I stumbled upon this post a little while ago. The essence of this visual is that it allows one to visualize the flow of data. The unique feature of this visual is that the width of each pipe represents the magnitude of each flow.

I even made use of this sort of visualization a while back when working on a project.

Sankey Diagram

I currently make use of this free budgeting app, 22seven, which links up with my account and provides me with a way in which to track my expenses.

Conveniently, the app provides the ability to extract your transaction information in a comma separated (csv) format.

I will be working with my friend on this as we both use this app and enjoy its abilities (by the way, this is the same person who worked on the previous project alongside me).


One Way

My house, located within the suburbs, faces onto a one way street. Ever since we moved in, we have always had trouble with drivers coming down the one way illegally. Now, there are cases where people do not know the area and do this without knowing. However, the rate at which people come up and down the one way, with the increasing population and traffic problems are making the situation worse. There is a stop street at the one end of the one way which people often do not stop at, never mind look up the one way to see if someone is coming down the one way. This situation is illustrated by the following image. The dotted line over here indicates the part where the road transitions from a one-way to a normal road.

</embed>

Naturally, as you can see, my house is relatively near the transition. So, technically, I could simply go down the one way. I don’t. I go up the road and around the triangular block to go in some directions. However, as you can see, by the time the cars get to this point on the road, they have picked up some significant speed; if they know they are in the wrong, then it is highly likely that they are trying to get out of that section as fast as possible as well. This, has not yet (to my knowledge), led to any accidents.

I am going to attempt to create a system to try and stop this. Or at least have some fun tracking these people going down the one way.

The advent of networked cameras has made this a possibility. All that I need to do is be able to run some cable down to the gate, set up a camera, and set up some sort of tracking system for these cars.

At the moment, I have created a repository to make use of for the process.

An added benefit to installing cameras down by the gate is that we will then actually have a camera down by the gate to see when visitors are outside (which is really convenient as they currently hoot at the gate).

There are also currently a number of machine learning algorithms (and many datasets out there for vehicles) that I will attempt to make use of in conjunction with OpenCV (I am going to try Python before I attempt to use Matlab) to perform this.


Converting ebooks

Calibre, the ebook library offers the ability to convert ebooks from and to multiple formats. It also has a CLI utility. There are a number of options for the conversion. The following command illustrates a simple conversion from one format to another.

./ebook-convert.exe "book.epub" "book.mobi" --pretty-print --enable-heuristics

Removing Sensitive Information from Git

There may be a case that you have a git repository that you wish to make public, or distribute to others. In addition to this, there may be sensitive information on this repository that you do not wish to distribute.

The first step, that I believe is the best procedure, if using github for the process, is illustrated in this post.

I will assume that your original repository is private and hosted on github. Next, I will assume that you want to keep that original repository private and work on it into the future. In this case, the best procedure may be to replicate this repository, and make this newly replicated and cleaned a public one (or easily shareable).

The Tools

You will need git and BFG on your system in order for this to work.

The Procedure

The first step is to replicate your original repository.

Cloning

Some of these instructions are taken from here. There are a number of options that this page allows, for example if your copied repository should keep updated with the original one (the procedure that I will be using is the first one).

  1. Make a new private git repository that you are going to be using as the “distributable” version.
  2. Clone a bare copy of that repository:

    $ git clone –bare https://github.com/exampleuser/old-repository.git

  3. Mirror-push to the new repository that you created

    $ cd old-repository.git

    $ git push –mirror https://github.com/exampleuser/new-repository.git

  4. Remove the temporary local repository you created.

    $ cd ..

    $ rm -rf old-repository.git

The next step is to find and replace/remove the sensitive files/information within the repository.

Cleansing

I will be making use of the BFG, which appears to be a better/alternative version to git-filter-branch.

The first step is to download the application, which can be found on the BFG website.

This is a java file, so make sure that you have the Java Runtime Environment installed.

The examples of how to use the program can be found on their website, however, take note that in order to run the application, make use of (take note of the file name that you downloaded):

java -jar bfg.jar

This will be the method on how you will run the bfg command as seen in the examples.

Now that this is out of the way, we can jump into the removal part.

This starts with first cloning the repository that you wish to cleanse:

$ git clone --mirror git://example.com/some-big-repo.git

Now you are ready to start cleaning up.

Passwords and Code

In order to replace text within the codebase, you can make use of a file with a list of the text that you wish to be removed/edited.

The best method to do this is to create a passwords.txt file with a list of the text that you wish to be removed/edited. The following illustrates the different ways this can happen (thanks to this guy, and some info from here):

PASSWORD1                       # Replace literal string 'PASSWORD1' with '***REMOVED***' (default)
PASSWORD2==>examplePass         # replace with 'examplePass' instead
PASSWORD3==>                    # replace with the empty string
regex:password=\w+==>password=  # Replace, using a regex
regex:\r(\n)==>$1 # Replace Windows newlines with Unix newlines

Take note, just like the files, you should have already removed the offending text from the repository, as these pieces of text will also be protected. One can use the command:

egrep -lRZ 'foo' . | xargs -0 -l sed -i -e 's/foo/bar/g'

This replaces ‘foo’ with ‘bar’.

Files

An important note, in the section for files from the BFG site, is that you should first delete your unwanted files and make a commit (and a push) before you run the BFG. The files (in the repo) within the latest commit are protected, which means that the BFG will not be able to delete the unwanted files.

There are cases where you may want to delete an entire folder. This can be done as follows (found here):

$ bfg --delete-folders "{folderA,folderB,folderC}" my-repo.git

Once this is completed, you have to navigate into the git repository.

cd some-repo.git

Then run the following command:

$ git reflog expire --expire=now --all && git gc --prune=now --aggressive

Once this has completed, you are ready to push the changes to this “ready for public” repository.

This completes the process of cleansing your repository.