Tweet Geolocations: $TSLA

I have been paying around with the R package twitteR.  It is an R interface to the Twitter web API.  I used it to search for the 1000 latest tweets containing $TSLA (Tesla's stock ticker).  I then removed tweets from outside North America, geolocated them, and plotted them on a map.  The map is below.

As we would expect, most tweets are from urban areas -- particularly from Washington D.C to Boston, Florida, and California.  Surprisingly there is only one tweet from Seattle, but quite a few from Atlanta.  Also, there are many more tweets from the southern states compared to the upper midwest.

When I get a chance (hopefully soon), I'll post the code/packages I used to create the map.

 

tsla

EIAdata R Package

My EIAdata R package is now available on CRAN.  This means you can install the package with a simple, install.packages("EIAdata") from within R.

A development version of the package is available on GitHub.  This package gives you programmatic access, from within R, to over a million energy related time series available through the Energy Information Administration's API.

CDs then USB won't mount: Debian sid (unstable)

A little while ago, my Debian sid system stopped mounting CDs.  Shortly thereafter it stopped mounting USB devices also. After searching around a bit I found a solution here (the problem was with Consolekit):  https://lists.debian.org/debian-user/2011/10/msg01252.html

The solution is to create a file etc/polkit-1/localauthority/50-local.d/consolekit.pkla with the contents:

[restart]
Identity=unix-user:*
Action=org.freedesktop.consolekit.system.restart
ResultAny=yes

[stop]
Identity=unix-user:your username here
Action=org.freedesktop.consolekit.system.stop
ResultAny=yes

of course replacing your username here. Also create /etc/polkit-1/localauthority/50-local.d/udisks.pkla with the contents:

[udisks]
Identity=unix-user:*
Action=org.freedesktop.udisks*
ResultAny=yes

Hope this helps if you run into the same issue.
 

Spectrwm

I use KDE as my main desktop environment, however I have been looking for something more lighweight which is focused on keyboard input versus the mouse.  I came across spectrwm [arch wiki], tried it out, and definitely like it.  Just using the default settings I rarely have to use the mouse -- see the Arch wiki for key bindings for basic tasks.   I also surprisingly like working with a tiling window manager.  It is available for both Debian and Arch. Amazingly, the installed size of spectrwm is 196 KiB.  Never has anything so productive taken up less disk space.

Note, while it would be natural to use spectrwm on a lighter system -- like my chromebook running arch -- the increased speed and ease of almost every task makes it competitive for full-powered systems.

Below is a screenshot of one workspace.  What is extremely useful, however, is that you can set up many of these workspaces and flip through them with 'Meta-5' (for the fifth workspace for example).

 

testscrot

 

 

 

Preparing the Acer c720 SSD for Arch Linux

I bought an Acer c720 Chromebook which I intend to use in much the same way as I use my smartphone -- if it works I'll get rid of the phone.

The Chromebook was $200, and has 2GB RAM and a 16 GB SSD.  It has an SD slot so you can add more memory.  For the price it is a great little laptop.  It comes with the Chrome OS, but I have wiped that OS and installed Arch Linux.  The install was fairly easy, though there is only one slightly tricky part which I note below.

There is good information about installing Arch on the c720 here:  https://wiki.archlinux.org/index.php/Acer_C720_Chromebook

Then also be sure to follow an installation guide like this one:  https://wiki.archlinux.org/index.php/Beginners%27_Guide

Now, regarding partitioning the SSD this is what I did (and was successful).  I used fdisk to create an MBR partition with a DOS partition table on /dev/sda. So first:

# fdisk /dev/sda

If you use p to look at the partitions present you'll see /dev/sda has 12 partitions. I first deleted all these partitions with d then 1 then repeat for all 12 partitions (d is delete and 1 is the first partition, so the next is d and choose 2 and so forth).

Once all the partitions are deleted you can input o to make the partition table DOS (this is the letter o) instead of GPT. Then use n to create a new partition. I just created one root partition (no separate home or boot) -- to do so you can just hit enter at the prompts. This parittion will be /dev/sda1. Then write the changes with w (you can check all is what you want with p before you write).

Lastly format the partition (I formatted it as ext4):

mkfs.ext4 /dev/sda1

At this point you can follow the installation guide.

Quarterly and Annual R Functions to Pull Time Series from the EIA's API

Below are functions to pull quarterly and annual time series from the EIA's API. The functions return time series of class xts.

You'll need the following libraries loaded (quantmod is just for charting):

library(XML)
library(plyr)
library(xts)
library(quantmod)

Then make sure you have your key:

key <- "[insert your key in quotes]"

If you don't yet have a key, see the earlier post on the function for getting categories.

The function for quarterly data is:

getQEIA <- function(ID, key) {

    ID <- unlist(strsplit(ID, ";"))
    key <- unlist(strsplit(key, ";"))

    url <- paste("http://api.eia.gov/series?series_id=", ID, "&api_key=", key, 
        "&out=xml", sep = "")

    doc <- xmlParse(file = url, isURL = TRUE)

    df <- xmlToDataFrame(nodes = getNodeSet(doc, "//data/row"))

    df <- arrange(df, df$date)

    date <- as.yearqtr(df$date)
    values <- as.numeric(levels(df[, -1]))[df[, -1]]

    xts_data <- xts(values, order.by = date)
    names(xts_data) <- sapply(strsplit(ID, "-"), paste, collapse = ".")

    assign(sapply(strsplit(ID, "-"), paste, collapse = "."), xts_data, envir = .GlobalEnv)
}

The function for annual data is:

getAnnEIA <- function(ID, key) {

    ID <- unlist(strsplit(ID, ";"))
    key <- unlist(strsplit(key, ";"))

    url <- paste("http://api.eia.gov/series?series_id=", ID, "&api_key=", key, 
        "&out=xml", sep = "")

    doc <- xmlParse(file = url, isURL = TRUE)

    df <- xmlToDataFrame(nodes = getNodeSet(doc, "//data/row"))

    df <- arrange(df, df$date)
    date <- as.Date(paste(as.character(levels(df[, 1]))[df[, 1]], "-12-31", 
        sep = ""), "%Y-%m-%d")
    values <- as.numeric(levels(df[, -1]))[df[, -1]]

    xts_data <- xts(values, order.by = date)
    names(xts_data) <- sapply(strsplit(ID, "-"), paste, collapse = ".")

    assign(sapply(strsplit(ID, "-"), paste, collapse = "."), xts_data, envir = .GlobalEnv)
}

Then, to use the function type:

getQEIA(ID = "ELEC.PLANT.GEN.13-WAT-ALL.Q", key = key)

Which will return an time series object of class xts named ELEC.PLANT.GEN.13.WAT.ALL.Q to the global environment (note the substitution of . for -). This time series is the number of megawatt hours of electricity produced by the Jordan Hydroelectric Dam per quarter. The time series is ready to be used by quantmod:

chartSeries(ELEC.PLANT.GEN.13.WAT.ALL.Q)

plot of chunk unnamed-chunk-6

Similarly for annual data:

getAnnEIA("NG.N3050MA3.A", key = key)

This is the annual natural gas citygate price in Massachusetts.

chartSeries(NG.N3050MA3.A)

plot of chunk unnamed-chunk-8

Note all of these functions pull the full time series history. However, since the resulting time series object is of class xts, it is very easy to return a subset of the time series. For example, if you just want the gas prices from 2005 onward you can use:

NG.N3050MA3.A["2005/"]
##            NG.N3050MA3.A
## 2005-12-31         10.64
## 2006-12-31         11.00
## 2007-12-31          9.34
## 2008-12-31         10.29
## 2009-12-31          8.29
## 2010-12-31          7.74
## 2011-12-31          7.04
## 2012-12-31          6.03

For more information on how to use xts series use ?xts in R.

 

 

R Functions to Interact with the EIA's Application Programming Interface (API)

The Energy Information Administration at the US Department of Energy has made a great deal of data available through an API. The data sets include 408,000 electricity series, 115,052 petroleum series, and 11,989 natural gas series. There are also 30,000 State Energy Data System series. The series can be browsed here: http://www.eia.gov/beta/api/qb.cfm

I have created a couple of R functions which will allow you to browse the data sets from with R, and download data directly into R as an object of class xts. Following is the function to browse the categories and find series IDs form within R – I'll post the functions which extract the data shortly. These functions are works in progress.

To use the functions you'll need to request a key from the EIA here: http://www.eia.gov/beta/api/register.cfm

You will need to have the following libraries XML and plyr libraries loaded.

library(XML)
library(plyr)

The function is:

getCatEIA <- function(key, cat=999999999){

  key <- unlist(strsplit(key, ";"))

  ifelse(cat==999999999,
         url <- paste("http://api.eia.gov/category?api_key=", key, "&out=xml", sep="" ),

         url <- paste("http://api.eia.gov/category?api_key=", key, "&category_id=", cat, "&out=xml", sep="" )
         )

  doc <- xmlParse(file=url, isURL=TRUE)

  print("########Parent Category########")
  tryCatch(print(xmlToDataFrame(nodes = getNodeSet(doc, "//category/parent_category_id"))), warning=function(w) FALSE, error=function(w) FALSE)

  print("########Sub-Categories########")
  print(xmlToDataFrame(nodes = getNodeSet(doc, "//childcategories/row")))

  print("########Series IDs########")
  print(xmlToDataFrame(nodes = getNodeSet(doc, "///childseries/row")))
       }

To use the function you pass your key, and the optional category ID. If you leave the category ID blank the function will return the top categories. For example:

key <- "[your key here]"
cat <- 40827

Then:

getCatEIA(key, cat)

returns

[1] "########Parent Category########"
   text
1 40203
[1] "########Sub-Categories########"
  category_id           name
1       40828           Real
2       40829 Current-dollar
[1] "########Series IDs########"
data frame with 0 columns and 0 rows

The result shows you the parent and sub categories. Note there are no series in the category. The series IDs are the identifiers for the actual data.

To see the root (top level directory) use:

getCatEIA(key)

which returns:

[1] "########Parent Category########"
[1] "########Sub-Categories########"
  category_id                            name
1       40203 State Energy Data System (SEDS)
2           0                     Electricity
3      456170                     Natural Gas
4      229973                       Petroleum
[1] "########Series IDs########"
data frame with 0 columns and 0 rows

To see series IDs we can choose:

getCatEIA(key, cat=40828)

which returns

[1] "########Parent Category########"
   text
1 40827
[1] "########Sub-Categories########"
data frame with 0 columns and 0 rows
[1] "########Series IDs########"
         series_id                                              name f
1  SEDS.GDPRX.OH.A                 Real gross domestic product, Ohio A
2  SEDS.GDPRX.MT.A              Real gross domestic product, Montana A
3  SEDS.GDPRX.NC.A       Real gross domestic product, North Carolina A

 units               updated 
 Million chained (2005) dollars 26-JUN-13 06.34.27 PM
 Million chained (2005) dollars 26-JUN-13 06.34.27 PM
 Million chained (2005) dollars 26-JUN-13 06.34.27 PM
< there are 52 total series here:  showing 3 for space>

You can use these series IDs (together with function I'll post shortly) to pull the data.

 

 

Extracting RSA Key from GnuPG Using Sound

I subscribe to the Debian security mailing list.  The advisories are usually about a buffer overflow in some package -- stuff that is good to know and is the product of hard work by people across the open source community.  However the recently released DSA-2821-1 was a little different:

"Package : gnupg
Vulnerability : side channel attack
Problem type : remote
Debian-specific: no
CVE ID : CVE-2013-4576

Genkin, Shamir and Tromer discovered that RSA key material could be extracted by using the sound generated by the computer during the decryption of some chosen ciphertexts."

That's right, the researchers were able to extract decryption keys from gnupg by analyzing the sound the CPU made during decryption. The news quickly made it to Slashdot here [new tab]. The Slashdot article links to the original research paper where the methodology is explained.