{ % feed_meta %} Home | A Few Pinches of Salt

Configuring R Markdown

I really love markdown. The simplicity and clean documents that it can generate. However, lately I have come across RMarkdown. A beefier version of markdown that can do so much more. It can run code, generate plots, including languages other than R which is cool.

But really, I wanted to create a document that follows the style of Edward R. Tufte, and some awesome people have put in the time to make a package that can do exactly this!

First, have R installed, this can be found here.

Then, just because it is more user friendly, install RStudio, it’s a nice light weight IDE that sometimes comes in handy.

Then, install the RMarkdown package inside R. If you want to do this inside of RMarkdown (or R), then all you will need is the following:

install.packages("rmarkdown")

Now, RMarkdown actually makes use of a number of other tools in order to generate the document that you want. Put simply, your ‘.Rmd’ file that you create will be fed into knitr, which will run all of the R code that you have in the file, and it will create a standard markdown file, this has the code and the generated output from each of the code blocks that you may have used. Next, and you are going to have to have pandoc installed for this next part, the compilation process uses pandoc to process the markdown file. This supposedly will generate your file.

Part of the process, naturally uses TeX, which is why, I think, the process generates .tex files.

Some way along the process of generating this document, it got stuck, after it had generated the .tex file.

So, as I already have MiKTeX installed on my system, I ran it there. It apparently needed to install a number of packages to get this to work, and MiKTeX generated the file that I wanted.

So, hoping that this was the reason that R was complaining, I ran the original script again, and voila, it generated the document without an issue!!

Anyways, the following line allows you to generate the document from the command line:

Rscript --default-packages=rmarkdown,tufte GenerateDoc.r

Where ‘GenerateDoc.r’ has:

#!/usr/bin/env Rscript

library("rmarkdown")
library("tufte")
rmarkdown::render('ProtocolDocumentation.Rmd')

The ‘.Rmd’ file also needs to have the following in its header:

---
title: Switching Protocols 
author: Tristan Kuisis
date: '`r paste("First created on Feb 24, 2020. Updated on", Sys.Date())`'
output:
  tufte::tufte_handout: default
  tufte::tufte_html: default
---

Kinda weird, lots of setup to run a single ‘.Rmd’ Anyways, it works now.


How to use gitignore to not ignore

I was recently working on a problem where I needed a git repository to have a folder that is empty. The reason for this was that the folder structure needed to be maintained such that when the repository was loaded into a new system, then dependent applications can easily place files within those already created folders.

Now, the standard way that repository folder structures work is that folders are not tracked if there are not any files within those folders.

More specifically, I want the system to ignore all files within the folder, but to still make the folder when I clone the repository.

Now, this solution doesn’t really circumvent the problem of having to have files within the folders in order to keep the folder tracked, however, it can be seen as empty. This simple method is to place a .gitignore file in the folder, then put the following code in the file (code found here):

# Ignore everything in this directory
*
# Except this file
!.gitignore

I also found this handy command in another answer for adding a .gitignore file in each of the empty folders within the repository:

find . -name .git -prune -o -type d -empty -exec touch {}/.gitignore \;

It’s also obviously had much discussion and is often needed based on the fact that it is one of the FAQ’s on the git website.


Garmin Sleep Analytics

When I got my Garmin watch near the middle of last year, I thought it would help me to better track my day and to optimize how I spend my time. Unfortunately, I eventually realised that the smart watch is only so smart as the feedback that it has programmed into it and how you change your habits based on what it is telling you. The watch has a number of sensors built into it, in general, it measures your steps and heart rate. The use of these sensors allows for a number of inferences to be made such as flights of stairs climbed, stress, and a number of other activities/analyses that are created in order to allow for greater insights. This guy has some useful information on how reliable each of these pieces of information are. This is purely based on the fact that some of these features are inferred from multiple levels down the data collection system.

Anyways, what I really wanted to do was to download my data in order to see what I can do with the actual data. In doing this, I saw that the bulk data download provides some really convenient JSON files that hold a few pieces of data on each day and how you slept. I created a GitHub repository to keep track of this process, it can be found here, it runs you through the process of downloading your own data and extracting it.

I just wanted to get an idea of how my sleep is compared to averages, or how much I should be sleeping.

Sleep Analysis and Research

There are numerous articles on the internet which speak about the best ways in which to sleep, how to organize your day to allow for better quality sleep, foods that you should/shouldn’t eat. I won’t cover these, instead leave this to the user and allow them to interpret their sleep data as they wish.

In order to provide some metrics or abilities to analyse sleep, I have looked at some journals that look into the science of sleep.

This article (and this one) has recommendations on the number of hours in a day that an individual should sleep based on their age. This is purely a recommendation and depends on the individual.

The SleepFoundation.org has some recommendations to attempt to get better sleep:

  • Stick to a sleep schedule, even on weekends.
  • Practice relaxing bedtime ritual.
  • Exercise daily.
  • Evaluate your bedroom to ensure ideal temperature, sound, and light.
  • Sleep on a comfortable mattress and pillows.
  • Beware of hidden sleep stealers, like alcohol and caffeine.
  • Turn off electronics before bed.

Some of these are not particularly suited to analysing with the data, however, I am taking note of some of these that may be easy to implement and test out.

This article, though not from a journal has some notes on what may be done to improve sleep. Take these with a pinch of salt as always.

Sleep Stages

In general, according to Wikipedia (and obviously whoever Wikipedia is referencing) sleep occurs two main stages, Rapid Eye Movement (REM) and Non-REM.

Each of these stages lasts for approximately ninety minutes, this means that in a normal sleep cycle, you will have 4 - 6 of these cycles.

The NREM cycle is broken up into three separate stages: N1, N2, and N3 where the last stage is often referred to as delta sleep or deep sleep.

The normal order of this is:

N1 –> N2 –> N3 –> N2 –> REM

The proportion of REM sleep is higher the closer to waking time. [1^]Interesting, I wonder if this is why I feel like I do most of my dreaming just before I wake up.


Logging Cron Output

Over the last few months I have really spent a lot of time working with Linux. Part of this work means that I have to set up automated systems that are able to log information, run scripts, gather data from sites, etc.

Luckily Linux comes with some really useful tools that allow for automated programs to be run. There are numerous configurations that one can use with crontabs in Linux which I am currently using.

However, when setting up these systems, I often ran into issues with how I configured the scripts, where they were stored, and how they were run.

In addition to this, cron doesn’t natively provide the ability to tell the user what happened when a cron job did (or didn’t) run.

The following template allows the user to set up cron tabs such that their outputs are placed in log files, such that debugging during the cron creation process and future failures can be captured for later debugging.

Users

The first issue that I came across was how the script is run and which user runs the script.

To access the crontab of the user currently logged in can be run with:

crontab -e

This opens the crontab file that has the list of cron jobs that are used in the machine and user 1 2.

Logging Output

Normally, you configure cron to email you when cron jobs fail, it also stores its logs in a distro specific folder.

I want to configure the cron tab so that it will send its output directly to a log file of my choosing, that way debugging can be made simpler and the logs can be directed into specific log files for each job. This cleans up the logging process as I can narrow down which process is causing issues without having to scroll through many entries.

The most simple way that I have found to log the output of cron jobs is the following:

*/1 * * * * /home/user/script.sh >> /home/user/script.log 2>&1

This will place the output from script.sh to script.log.

  1. Note: The cron job will be run as if it were being run by the user that the cron tab file is in. This means that it will run with the permissions given to that user, in addition to the path variables available to that user. 

  2. Also take care about where the cron job is being run from, this should guide you in how to format the relative paths that you use to run the different programs. 


Automated Electric Fence Status Monitoring

Since we installed an electric fence at home, we have had endless problems. This post is a follow-up from my first attempt at solving the problem in One Way Tracking Part 2. The fact that it is a home installation means that we are the people who need to maintain and fix the fence every couple of months. In addition to this, whenever it rains, the electric fence alarm has been set off. This is often due to leaves or branches of nearby trees getting caught on the fence and setting the alarm off. Sometimes it is due to dirt forming on the surface of the insulator from the wire to the rod keeping the fence up.

Some of this is avoidable with consistent trimming of the trees nearby, however, it is not always possible and the alarm does go off.

We have also experienced times when the fence simply turns off 1.

The Problem

Now that we have got an idea of what is happening, I needed to figure out how best to tackle the problem. Basically, it stems from the issue that the electric fence module turns off the electric fence output and will either alarm 2 or simply be off. This can be for a number of reasons (of which I won’t get into).

This leaves the issue with two possible cases:

  • The electric fence is in the alarm state
  • The electric fence is off.

In both of these states, the output to the fence is off. So, how do I solve this.

The Solution

There are a number of methods of approaching this issue, the electric fence provides the ability to install an add on module that has the capability of communicating with a security service 3. I do not like this approach as it would require the company to come out and evaluate the fence 4. In addition, this would cost money to keep running.

Barring the use of a product developed by the company, I need a way of monitoring the system myself. This means designing a solution that will be capable of monitoring the status of the fence at any given time and providing me with updates on the fence if it changes, the main goals can be as follows:

  • Continuously monitor status of electric fence output
  • Reliable monitor status
  • Provide notifications on electric fence status changes (alarm or off states)
  • Automated system of monitoring
  • Automated system of notifications
  • Monitoring from anywhere

Simply put, the system needs to be able to reliably and continuously monitor the status of the electric fence periodically 5.

The last requirement stated in the list implies that the system needs to be able to provide the status of the fence from anywhere, which means the information should be accessible from the internet. This is particularly useful in cases that I am not at home and I am not connected to my intranet with direct access to the monitoring system.

The Design

In the previous post where I talked about this process, I since figured out that there is an additional LED that is a direct indicator of the status of the fence. This is the LED corresponding to the “ON” status. This is an LED that conveys the status of the fence at any given point in time.

In that previous post I stated that I could make use of a camera to continuously take images of the face of the module and do some form of computation on the image to determine if the LED is on or not. This is a fairly complex solution that will required setting up a camera at the fence 6.

I thought of another method of detecting the fence status which is designing a measurement device that measures the voltage of the fence, however, measuring this high voltage with simple and low-cost equipment didn’t appear feasible. In addition, I came up with a far simpler solution to the problem.

I knew that I could tell the status of the fence depending on whether the LED was on or not, so I simply had to design something that could tell me if the LED was activated.

I could do a measurement of the voltage across the LED 7 to determine if it is on or not. The difficulty with this method is that I would have to create a solution to get the measurement into the box while keeping the lid closed. The lid is used as an interlock to make sure that if the lid is opened, the fence immediately disables its output and turns off.

I didn’t want to be soldering onto the board or drilling into the box which would probably void the warranty and also be a hassle in the case that the module broke and had to be replaced, meaning I would have to start the process all over again.

So, I used a tried and trusted technology, the simple light dependent resistor (LDR). This simple electronic device changes its resistance depending on the light incident on its surface.

The LDR

The light dependent resistor adjusts its resistance depending on the level of light incident on the surface of the component. So, all that I need to do is to place the LDR onto the surface of the fence module and measure its resistance. This resistance is an indication of the state of the fence at any given point in time.

Put simply, the LDR can only be in any of two resistances. Each resistance corresponds to either the ON state of the electric fence or the OFF state of the electric fence.

Measuring the LDR

One cannot simply place a multimeter on the LDR each and every time I need to get the status of the electric fence, so I have to create a simpler method of monitoring this value. Fortunately, I can make use of one of the first circuit analysis tools any person is introduced to, the voltage divider.

Put simply, I will place a resistor in series with the LDR and put a voltage across the two components. Measuring the voltage across the point where the two devices interface will give me an idea of what the resistance of the LDR is, as I already know the resistance of the normal resistor.

All that this now needs is a voltage source and a voltage measurement, which is a lot easier and straight forward to monitor with the use of an Arduino.

The Arduino here is used as the 5 V reference voltage and then the simple use of the ADC built into the Arduino is used to measure the voltage divider circuit 8.

The system is currently set such that the LDR is connected to ground and the resistor to 5 V, this means that the voltage measured should be at one of two levels.

The exact voltage level does not matter much as it will be either a “high” or a “low”, I calibrated this during testing to know which range I should provide for the monitoring and status notifications. Essentially, if the voltage measured goes above a specified threshold, then I will set off the notifications.

The Communication Part

At this point I have an accurate method of continuously monitoring the status of the fence, however, I need a method of getting this information from the fence to the internet. Luckily I had access to an ESP-01 chip from a few years ago which provides the ability for the Arduino to connected to a Wifi network and communicate with the internet. I simply used this to communicate with a ThingSpeak channel and continuously send the voltage measured through to the internet through this channel.

This is a relatively simple process to get this value out to the greater world as I can continuously send the value through to some database on the internet.

The Notification Part

Now I need to create a system that is capable of notifying me of any changes in state of the electric fence.

I love the system that IFTTT has created as they have the ability to interface with ThingSpeak. This means that all I need to do is to connect my IFTTT account to ThingSpeak and create a recipe in ThingSpeak such that it will trigger the IFTTT recipe. This allows the recipe to send an email notification to my main email where I can see the status.

The Aftermath

Over the past couple of months, I have had this system up and running. It runs relatively flawlessly and provides me with an up to date status of the electric fence, giving me emails when the fence is off. I have successfully used it to notify my family when the fence turns off, even when I am away.

The only problems that I have encountered is my method of attaching the LDR to the face of the electric fence module. I have used duct tape to attach the LDR to the module and over time, the tape appears to lose its adhesiveness to the surface. Naturally this will happen as this piece of equipment needs to be rugged as it is outdoors and experiences extreme temperature swings, rain, wind, dust, etc.

I still have yet to find a failsafe method of keeping the LDR in place.

So far this system has performed quite well, providing timeous and reliable status updates of the electric fence when required.

  1. None of this is what you want happening with an electric fence as sometimes the alarm does not go off and we end up going about our lives without knowing that the electric fence has been off for long periods of time. 

  2. Silently or not, depending on how the fence is configured. The way that it is currently configured is that upon each power cycling, the alarm is turned on or off. 

  3. The electric fence module is a Merlin 4 from Nemtek, I have found two of their operating manuals online here, and here. There appear to be a few different versions of the system. 

  4. This also means that each time out fence is damaged, we would have to call out the repair crew which would cost money and generally be a hassle when we could simply fix it ourselves. 

  5. This parameter can be changed depending on requirements. Five-minute intervals should be more than sufficient to provide timeous information of the system. 

  6. At the time of writing, I have not been able to get a camera down at the gate yet. 

  7. And I suppose its series resistor. 

  8. The Arduino is powered with the use of a power bank; this helps for when the power goes out at the gate as the Arduino can run for an extended period of time.