Flattening Python Code with "continue"

Flat code is better than nested.  It's more readable, easier to update, and easier to debug.  All of the linux kernel source code uses 8-space indents because, as I've seen it explained by Linus, if you can't use enough indents with 8-space, you have too many indents and need to refactor your code so it's more simple.

I do some data analysis type stuff at work fairly often.  Sometimes I'll have a half-dozen conditional statements to satisfy in a loop where I'm ticking over rows of data.  Using the "continue" statement is a really simple way to keep code flat and manageable.

Bad form would be like this:
for row in my_data:
    if row[val] == 'blah':
        if row[val2] == 'test':
            doFunc(row)
            doFunc2(row[val3])
            if row[val3]:
                yield row


Better would be like this:
for row in my_data:
    if row[val] != 'blah':
        continue
    if row[val2] != 'test':
       continue
    doFunc(row)
    doFunc2(row[val3])
    if row[val3]:
        yield row

Raspberry Pi Projects

For years, I've been listening to people talk about the projects that they want to use the Raspberry Pi boards for.  I've never really had any good ideas... until now!

Automotive Data-Logger

This would include a dash cam and a reverse cam and would be a fairly involved project.  The RPi is a great candidate for this project because of it's low power requirements, hardware compatibility, and just because it runs Linux and is easy to program for!

What I envision is a fully contained unit that affixes to the ceiling in the cab of my truck up by the rear-view mirror.  It would:

• House front and rear-facing cameras.  The RPi has a special connector for a specific camera that works well with it, and there's a cable splitter available to enable connecting more than one camera.  Would use this.
• The Pi board
• Sensor accessory board (has accelerometers and stuff)
• Time/clock accessory board - this enables the RPi to keep track of time even when it's powered off, which is apparently an issue since it doesn't have a battery built in. 
• Bluetooth and 802.11 b/g/n dongles.  It might be reasonable to even use an external monopole antenna mounted atop the truck for better reception. 
• A notebook HDD
• USB GPS dongle
• A backup battery

I'd have to write some software for this one... I'm unsure if I could throw it together in bash or if it would be complicated enough that I'd want to do it in python.  Software Features would include:

• Event Awareness:
• Automatic detection of car starting to trigger logging. 
• Detection of poweroff to end logging
• Sudden stops trigger additional logging? 
• Logging while vehicle is on:
• Front and rear video
• GPS data, as available, but expectedly including position and velocity. 
• Accelerometer data
• Log wifi access points observed, signal strength, etc (wardriving)
• Network Services:
• Seek out and connect to open wifi networks 
• Auto-connect to home network and automatically sync logged data over to home server. 
• Tether from phone for internet - look at usb devices connected and auto-start the tether software when phone connects... could even adb issue command to the phone to start the tether software!
• Share available network via LAN to other devices in the truck...  For example, if I ever get around to building a Nexus tablet CD Deck replacement, it would provide a network connection to it when available.   

References - there are lots of these builds online:
http://pidashcam.blogspot.com/
http://dreamgreenhouse.com/projects/2013/picar/

SDR Radio Server

This is a pretty simple one.  Just install linux to the Pi, install the SDR drivers and software, and configure it for the network.  SDR servers allow clients to connect and control the frequency that the SDR radio listens on and, via the network, listen to the radio.

The SDR receivers a lot of people use are cheap, like $20, and work on all kinds of bands, from HF and shortwave, up through the FM stations, VHF, UHF, and beyond.    They are limited by their inability to filter signals received from the antenna, so powerful FM broadcasts can pollute the rest of the spectrum without filters in front of the receiver.  Of course, having a quality antenna is really important as well.

Ideally, I'd have a random wire antenna set up somewhere on my property, running into a window or something, and the RPi would serve the radio over the local network so i could listen from anywhere at home, w/o a tether.  It'd make it easy to listen to SW broadcasts.

Property Invesetment - Finding ROI

So I've been looking at buying a duplex or something and am trying to understand what is actually a good investment.  I put together a spreadsheet, here, that allows values to be plugged in and it will show the annual increase in value and %return on total investment in the property.

Plug in values in yellow and look at cells in green on the right for return.  Red shows my cost each month toward the mortgage.

It includes mortgage insurance automatically for down payments of less than %20 of the total value, and then stops them once the M.I. cutoff has been reached, configurable in yellow.  I'm not sure if I'm dong this right, but it looks good to me :p

Outlook 2011 Mac OSX Problems

Just some hints for others incase them run into similar issues.  I went on vacation last month and when I returned, Outlook would crash constantly.  Local IT support recommended that I rebuild my profile and, second, try removing my Exchange account and rebuilding the profile again.

This didn't do anything, and somewhere along the way, all of my local folders, the ones that show up under "on my computer", disappeared.  I was still seeing crashes and I had time machine backups, so I charged forward anyway.  I've heard reports of problems resulting from the 14.5.0 update.

What fixed the crashing for me was to completely reinstall Office 2011 (including Outlook).  Google around for the Office Uninstall Tool.  Microsoft provides one.  Reinstall worked fine, although the Update tool had a hard time and I had to grab updates from the Microsoft site to get Office and the Update Tool up to a recent version.

I was hoping I'd be able to try and use an earlier Outlook release, in case one of the updates that my system grabbed when I first got back from vacation had caused my issues, but it complained that my Profile was only compatible with current outlook releases, so I went ahead and updated to the most recent 14.5.2.

My "On my computer" folders were still missing!!!  I restored my profile from a few days ago using a time machine backup, and still nothing.  I started digging through settings to see if I could find a way to load them manually, and something happened that fixed it; it was one of two things:

• I tried to do an export of all of my contacts, calendar, mail, to a .olm file, just in case, but ended up canceling it because it was too slow (I was trying to write to an SD card where I dump general data).  
  
• It's possible that I accidentally clicked the check box in the Preferences -> General -> Folder list -> [ ] Hide On My Computer folders.  By this time I noticed this, I had been through settings once... this may be all that I needed to make these folders show in the first place, but who knows.  If I check it now, they disappear!  
  

There wasn't much good info coming up in Google about my problem with the local mail, so hopefully this helps someone!

Notes on OSX Virtalization, booting raw disks

I picked up a Mac to use for work, primarily, a couple weeks ago.  It's a late 2013 Macbook Pro Retina 13" w/ 8GB ram and 256GB SSD.  I wanted to be able to dual boot with Linux for home and dev type stuff, but being short on time needing to encrypt my primary hdd, I didn't get to figure out boot camp before finalizing my setup for work, so I'm not about to mess with my primary disk's partition/efi config and risk downtime at work.

Sooo, I'm trying to get set up to boot to boot Linux on an SD card.  This has been a bumpy road.  I purchased a Nifty SD card adapter and a Sandisk 80MBps 64G card to install to, and I'm doing the install via a VM while I work on other stuff.  This presents first problem - support for booting SD card by Parallels, Fusion, or VirtualBox.

I'm defaulting to Parallels since everyone else on mac uses it at work, but Fusion looks pretty comparable.  I've used VirtualBox a lot in the past, but I got tired of bugs and crashing at some point and switched to VMWare workstation at home.

That said, VirtualBox seems to be the most flexible and capable when it comes to booting SD cards on Mac.  It is the only virtualization solution of the three that doesn't have  "dumbed down" interface for people who expect things to just work without understanding them on mac.  It comes with command line tools for creating virtual disk files, etc.  This is what you need if you want to make a "bootcamp" disk image to use with Fusion.

Parallels and Fusion only support booting raw disks via "bootcamp".  You can't just point them to a block device and have them use it without tricking them.

• To trick Fusion into booting to the SSD, set up a linux VM with a virtual disk that you'll replace.  Then use virtualbox command line tools to provision a raw disk vmdk and copy this over in place of the virtual disk that Fusion created.   That's it. It should boot fine. 
• To trick Parallels, and note that I haven't found a way to do a raw disk vmdk like with Fusion, create a VM without a disk and enable efi.  You may want to remove Network from the boot order.  In the VM settings menu, attach the SD card to the VM.  Power up your VM and wait for it to drop you to the efi shell.  Then run:
• fs0:
• cd EFI\ubuntu
• grubx64.efi

Getting the macbook to boot the SSD is another matter, and it's where I'll look at the actual Ubuntu build.  

Macbook Pro Dualboot Ubuntu SD Card & Virtual in OSX

This is just a quick post outlining how I finally was able to get ubuntu installed on an SD card that would boot from the option menu when I power on the Macbook and would also boot in either Fusion or Parallels in OSX.

For reference I'm on a Q4 2013 Macbook Pro 13" Retina with a SanDisk Extreme Pro 64GB SD card.  Not the fastest, but nearly the fastest.  I finally had success using VMWare Fusion Pro 7.0.  This may work with Parallels, which seems to have better EFI support, but has not been very stable for me.  This solution would not boot in Fusion 6.x.

• Install the SD card in the laptop.
• Use VirtualBox to create a .vmdk disk file that points to the SD card (/dev/disk2, in my case).  This effectively creates a "bootcamp" disk image to use with Parallels or Fusion. 
• Download the rEFInd (the more recent version of rEFIt) usb stick disk image and dump it to the SD card:
• Disk Manager to see what the device name is for the SD card.  It was /dev/disk2 for me. 
• Use mount to see which partitions of the SD card are mounted:
• mount | grep /dev/disk2
• blah blah /dev/disk2s1...
• Use diskutil to unmount:
• diskutil unmount /dev/disk2s1
• ...
• Write the rEFInd image to the card:
• dd if=rEFInd_blah.img of=/dev/disk2
• Use diskutil info to take note of the name of the disk2s1 partition name. Something like ElTorito. 
• Download your choice of linux distro.
• Create a VM in VMWare Fusion and:
• Attach a 20GB IDE hdd to it.
• Attach the Linux install CD and set it as default boot. 
• In the VM config directory, remove the 20G disk image vmdk files and rename the .vmdk that was created in VirtualBox to match. 
• Add firmware="efi" to the vmx config file for the VM. 
• Boot the new VM and when it loads into the Linux Live CD:
• Dump /dev/disk2s1 (this is the 4.5MB partition that was created when we dumped the rEFInd image file to the card) to a tmp file:
• dd if=/dev/sda1 of=./sda1.dd
• Use gdisk (the GPT equivelant of fdisk) to:
• Take note of the starting and ending sectors  and the partition type of the refind partition (sda1)
• Wipe the GPT table (under expert optoins somewhere) and then re-open gdisk.
• Create a new partition of for rEFInd using the same starting and ending sectors we used earlier and the same partitoin type. 
• Set the name of this partition to the same as it was before (ElTorito?)
• Create a linux partition using the remaining space.  You may want to leave some buffer space after the first partition in case you ever need to update refind and don't want to have to resize or rebuild the linux partition. 
• save and exit gdisk
• Restore our image of sda1 to the new partition we created along with the new GPT partition table:
• dd if=./sda1.dd of=/dev/sda1
• Reboot the VM with SD card as default and verify that it comes up in the rEFInd boot menu. 
• Reboot the VM and boot to the CDROM again and start the install. 
• I'd recommend using ext2 filesystem, since it doesn't use journaling and is fast, and no swap, to conserve writes to the SD card.  
• Tell Ubuntu (or your distro) to install the bootloader to sda2!!! this is critical, so it doesn't bork your rEFInd bootloader.  
• When the Linux install completes, reboot the VM to SD card and verify that rEFInd loads and that, from there, you can select the linux image and boot to it.  
• if that works, then reboot the laptop and hold "option" to get the boot menu.  You should see the SD card and be able to boot Linux via rEFInd!

So Simple!    You don't want to know how many days of tinkering and repeated linux installs it took to find a setup that would work like this.  Fack!

Trying to boot to CD and install ubuntu directly to the SD card never worked for me, as the MAC bootloader wouldn't see the installation and present it in the boot menu when the laptop powers up, despite Ubuntu installing EFI boot junk on the SD card.  It wasn't too difficult to get the SD card to boot in EFI or legacy mode inside of a VM once figuring out how to do the bootcamp vmdk, which raises another important thing to note:

Fusion and Parallels are playthings only.  They really have no advanced tools included via command line or the standard UI.  There's no way to "bootcamp" to an SD card with them without hand-editing their config files and using 3rd party tools to create the disk image files.  VirtualBox is indispensable when manipulating VM images.  It comes with a bunch of great command line utilities for creating and managing disk images, and who knows what else.  I don't use it normally, though, because I used to run into issues with stability when I used it in Linux.  Meh... It worked great before Oracle bought it.  :P

Update:  Just saw the following, about creating raw disk iamges with vmware tools... might be helpful:
http://techrem.blogspot.com/2012/12/add-physical-disk-to-vmware-fusion.html

And this, which talks about using "tmpfs" to redirect IO-intensive filesystem activity to ramdisks and cut down on SD card activity:  http://www.zdnet.com/raspberry-pi-extending-the-life-of-the-sd-card-7000025556/
He suggests:

tmpfs    /tmp    tmpfs    defaults,noatime,nosuid,size=100m    0 0
tmpfs    /var/tmp    tmpfs    defaults,noatime,nosuid,size=30m    0 0
tmpfs    /var/log    tmpfs    defaults,noatime,nosuid,mode=0755,size=100m    0 0
tmpfs    /var/run    tmpfs    defaults,noatime,nosuid,mode=0755,size=2m    0 0
tmpfs    /var/spool/mqueue    tmpfs    defaults,noatime,nosuid,mode=0700,gid=12,size=30m    0 0

Might be useful... but my system wouldn't boot with all of these enabled... will have to actually think about this and do some testing later.

Bash Love - Simple remote connections

Bash is my go-to language for busting out sysadmin stuff.  I've been ramping up with python for a while now, and for more complicated tasks, it's fantastic;  it should probably even be fantastic for simple tasks.  But, there is one thing that I do all of the time in Bash that I have yet to find a way to replicate without unnecessary complexity in python:

#!/bin/bash
agregateResults (); { while read line; do echo "Program Output: $line"; done; }
clientCode (); { echo "I am client $(hostname)."; }
if [[ "${1}" == "client" ]]; then
    clienCode
else
    for server in  ${*}; do
        cat ${0} | ssh ${server} -s client
    done | agregateResults
fi

This is so freaking useful.  I have servers across multiple domains and I need to run script to aggregate data from all of them, but I can't always count on network file systems being available, etc.  This lets me have a script in once place and execute the same code everywhere without fumbling with copying it ahead of time, etc.   It's so clean and easy.

When I try to find out how to do this with Python, I read all kinds of stuff about parallels, Paramiko, blah blah blah... it's all so complicated.  There has to be a five-liner example out there somewhere of how to do the same kind of thing.  

Started a code-golf at stackexchange for this one: http://codegolf.stackexchange.com/questions/20008/self-execute-over-ssh

Pain with vCenter 5.1

I'm wishing I could just use 5.0, but I need some of the 5.1 functionality for nested virtual servers...

After running the Simple Setup, I get an error trying to log into the web client:

Failed to connect to VMware Lookup Service
https://:744/lookupservice/sdk - SSL certificate verification failed. 

Rebooting and waiting ten minutes for the service to start up fixes this.

After successfully logging into the web client with admin@System-Domain account, it doesn't list any available vCenter servers.  It appears that the vCenter service never registered with the Lookup Service, so this has to be done manually, per the doc at http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=2033620.  See the "Registering vCenter Server with Inventory Service" section.

cd C:\Program Files\VMware\Infrastructure\VirtualCenter Server\isregtool

register-is.bat https://machinename.corp.com:443/sdk https://machinename.corp.com:10443 https://machinename.corp.com:7444/lookupservice/sdk

This seemed to work well.  Loads of text scrolling by during the register.  After connecing back up to teh Web Client, There was a message showing the following (not exactly, I didn't capture this verbatem):

Unable to connect to one or more vCenter Servers:
https://:xxxx

So I'm pretty back to where I was before trying to register the vCenter server with Inventory Manger.

Linux Swap Create Init Script

I use an SSD as my primary drive.  I used to have a swap partition on it (I know, bad idea on SSD) and it had probs w/ data corruption and the swap randomly being disabled.  Linux supports trim for SSDs, but trim won't do anything for swap that's always on a specific set of blocks in the SSD...

If we need swap space, but don't want it to always hit the same part of the SSD, we should be able to use a swap file in /tmp, allocating 2GB at boot. At least in Ubuntu, /tmp is cleaned up each boot, so when we create our swap file, trim has the opportunity to put the file on some fresh SSD sectors.

Here's the init script for this:


#!/bin/sh
#
# setup swap file init script
#

### BEGIN INIT INFO
# Provides:          atop
# Required-Start:    $syslog
# Required-Stop:     $syslog
# Should-Start:      $local_fs
# Should-Stop:       $local_fs
# Default-Start:     2 3 4 5
# Default-Stop:      0 1 6
# Short-Description: Monitor for system resources and process activity
# Description:       Atop is an ASCII full-screen performance monitor,
#                    similar to the top command, but atop only shows
#                    the active system-resources and processes, and
#                    only shows the deviations since the previous
#                    interval.
### END INIT INFO

PATH=/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin
SWAPFILE=/tmp/swapfile
NAME=init_swap.sh
DESC="tmp swapfile creator"

case "$1" in
        start)
                echo -n "Starting $DESC: "
                if $0 status >/dev/null
                then
                        echo "    Already Running."
                        exit 0
                fi

                test -f ${SWAPFILE} && rm ${SWAPFILE}
                dd if=/dev/zero of=${SWAPFILE} bs=1M count=2048
                mkswap ${SWAPFILE}
                swapon ${SWAPFILE}
                echo "        Done."
                ;;
        stop)
                echo -n "Stopping $DESC: "
                $0 status >/dev/null && swapoff ${SWAPFILE}
                test -f ${SWAPFILE} && rm ${SWAPFILE}
                echo "        Done."
                exit 0
                ;;
        status)

                if swapon -s | grep -q ^/tmp/swapfile

                then

                        echo "Swap is enabled"

                        exit 0

                else

                        echo "Swap is not enabled"

                        exit 1

                fi

                ;;

        *)

                N=/etc/init.d/$NAME

                echo "Usage: $N {start|status}" >&2

                exit 1

                ;;

esac

exit 0

Way cool dream about freedom of information...

I've had a few remarkable, vivid, dreams over the years.  This is one of them.

I find myself sitting in a college lecture hall, much like the old ones at Shasta College. There are two people up front, a man and a woman; I’m not sure which of them was actually the instructor. They seem more interested in each other than teaching and put on some music, using an iPod, that they want the class to listen to. It’s hard to hear and the class gets restless.

People in class start suggesting better things to listen to. I’m tired and slump down in my chair, half asleep. My friend Jon, sitting diagonally behind me says we should listen to something. I say lethargically that I can’t even hear him. He starts the song playing on his laptop.

As I glance over, I see the album cover on the screen and I’m pulled into the music!

I find myself on a floating airship, totally steam-punk style. It’s an elongated oval hewn from yellow wood, with two great propellers spinning underneath, one at each end, and the ship is probably fifteen feet long. Offset from the perimeter of the craft by a small walkway is a bookshelf that wraps most of the way around and is loaded with books facing outward. All of the ship’s wood paneling is carved with intricate symbols.

Listening to the music, I look into a gap between a couple of the books, and it seems that the shelf is forever deep, despite the ship only being a few feet across.

Glancing around, I see that there are four well dressed and well mannered fantasy creatures, each a different species having a variety of ears and humanoid/animal faces.

As the music plays, the creatures chatter about how the Marxists, the Librists, and the Serialists are no longer being persecuted and that they can finally return to society.


INTERPRETATION:

The school scene is symbolic of the poor state of our educational systems, or perhaps any social program that’s in place that perpetuates the state of our society. The teachers work to satisfy their flesh while they do a crappy job of feeding the class whatever the system has given them to teach.

The students represent society. They are fed so much BS through the schools and the media that, although they know that there’s a problem, most can’t figure out what to do about it.

The magnificence of the airship stands for a golden age that will come when information flows freely. The truth speaks for itself, and dysfunction cannot be hidden by bureaucracy. The books are the wealth of information that is unavailable to us.

The persecution is our world’s corporate and political attacks on free speech, whistle blowing (reform), and on file sharing.

The Marxists bring social change or reform. The Librists are keepers and sharers of knowledge. The Serialists are those who digitize and preserve information.

That the persecution is over gives me great hope!