Hey, you sass that hoopy Bill Napier? There's a frood who really knows where his towel is.

Two Weeks with Android Auto

The Beginning

Trust in the Waze

About a year ago I became a dedicated Waze user. I used to go everywhere, including to work each morning, a drive I've done for 6 years now. Margaret used to give me grief about needing GPS to get to work, but it wasn't really the directions I needed. Waze constantly looked at all the routes home and their current traffic conditions and told you the fastest way to get there. This is invaluable for a daily commuter, as Waze will route you around random slowdowns, huge accidents on 101 that you didn't know about, etc.

After a few weeks it quickly became apparent that glancing down at the seat next to me to see what Waze was telling me was going to end badly, so I shopped around and found the mount pictures below to hold me phone.

My Dashboard

It worked really really well. It held my phone in place and I could see what Waze was telling me. In addition to just Waze, I have a bluetooth device that plugs into my cars CAN network and is able to report sensor data. Install Torque, and you have a fancy new extended dashboard. Bundle in a dashboard app to hold your widgets, some Tasker automation, and you've really got something.

The downside with my homegrown setup is that I outgrew it and got tired of maintaining it. Especially since I still spent most of my time in Waze, so all those nice other widets weren't getting used.

Enter Android Auto

I decided to try Android Auto, Google's foray into the car. This didn't work with my factory headunit, so I had to buy (and have installed) a Pioneer AVH-4100NEX. This unit is awesome even without Android Auto, and the way things got installed I could actually replace Android Auto with my homegrown setup again via MirrorLink.

Android Auto

What Does it Do?

In short, it brings Google to your car. Of course you can do simple things that you would expect, like use Google Maps for navigation or listen to music via Google Play Music. But there are two areas where it really shines.

Google Now

Andoid Auto leverages the same technology that brings you the really useful before you knew you needed it Now Cars to your dashboard. When the system starts up, you're presented with a list of cards depending on what your recent actions have been. Searched for an address on your phone recently? You'll see a card with travel time to it and a button to hit to start nav. If you've made or received any calls recenlty, you'll get a contact card for the person, in case you want to call them back.

As things stand now, the cards are really rather limited, basically the minimum they could be to be useful. I expect over time Google will make them more and more useful, especiallly once we start seeing some 3rd party cards show up on the dash.

Voice Command

Ok Google

Hit the button on your steering wheel control and you get that familiar Google Now beep.

Navigate home

And you're on your way home.

Listen to 90s Frat Barbecue Radio

Now you've got some tunes.

Send a message to Margaret

And you can dictate a text message without removing your hands from the wheel. And if you get a text message, a toast pops up on the screen that you can hit and Android Auto will read it to you.

How Does it Work?

Google realized something very early on: The car companies move slow. Like really really slow. Some quick Googling indicates that you could easily get an Audio CD player in 1982. It took until 1987 before Ford offered the first OEM CD player in a car. How can you deal with an industry that iterates this slow?

The problem was solved by making the headunit in the car dumb. When you plug your phone in, the USB cable is used to project images/video generated on the phone. In addition, after the original set of features landed in Lollipop (5.0), the rest of the changes could be rolled out via individual app updates via the Play Store. This is a really elegant solution to an ugly problem.

What's bad about it?

My biggest complaint would probably be having to physically plug the phone in every time I get in the car. But it's no worse than my homegrown system, since I needed to plug the phone in to get power. So while annoying, not a show stopper.

Next in line would be the limited (maybe 10?) apps in the ecosystem that work with Android Auto Much like Android TV, it's been hard to get people to develop for it because it has such a small install base (And I think most of them work for Google). But I'll be cautiously optimistic that it will start to take off like Android Wear is now once more Android Auto supported cars are on the road.

The ecosystem problem boils down to a Chicken-and-Egg problem. You don't have any cool apps because you don't have that may units being sold. You don't have very many units being sold because you have no cool apps.

Should I Get It?

Ultimately, the answer boils down to "not yet". For people not buying new cars, it's just not yet a compelling enough system to shell out for the new headunit and install, unless you were looking to do that anyway (maybe your factory one is broken or lacks bluetooth).

And if you're in the market for a new car, the answer above still applies. There are so many other criteria that you should be looking at when buying a car other than if it supports Android Auto. Especially since Android Auto is only available in an OEM system in a very limited number of cars.


It will get better over time, hence the "not yet" above. Here's some reasons why:

  • Due to a smartly designed architechture, the Android team can quickly iterate and push out new features.
  • More and more OEM units will support Android Auto, obviating the need for an aftermarket headunit.
  • More apps will come.

Powered By Kubernetes

I had tried this in the past, but at the time Kubernetes (And Google Container Engine) were too immature to actually run something on. I checked in again a few months later and things were a lot more mature, but still very alpha.


I host a few blogs for people (this one included) and for the past few years have been doing it on an EC2 instance. I wasn't happy with how things were setup, mainly because I had invested so much time in getting this setup originally, that I wasn't looking forward to repeating that process to move someplace else (like Google Compute Engine).

Enter Docker. I started converting my hosting things into their own containers, with the hops of making things easier to test/run and to move to a new provider. But Docker had pain points as well. I still needed to do a lot of work on the host side to keep the containers running (mainly due to docker not wanting to auto-restart my failed containers, or being albe to figure out linked container dependencies).

Thankfully, Kubernetes is a lot nicer and since I was able to use GKE (Google Container Engine), I didn't have to do any mucking host side. Just write a few config files, hand them off to Kubernetes, and everything is running.

Pain Points

It was not all roses. Probably my biggest pain point was documentation. And not the typical "there is no documentation" or even "the documentation is old". In this case Kubernetes and GKE were documenting different versions of Kubernets, and their velocity is such, that there wasn't much the same between them. This was a large source of confusion for me until I figured out which docs to trust (use the source, Luke!). This was especially bad when I couldn't get simple things like their example configurations up and running.

The rest of my pain points are kinda picky. I include them to not discourage people from using Kubernetes, but rather to illustrate how Alpha the software is. They will eventually fix almost all these issues, but they just don't have time yet.

  • Once you configure a cluster on GKE, it's that size forever. No adding machines, no dropping machines, no resizing machines.
  • The Kubernetes master has to run on the same class of machine as all the worker nodes. Since it's basically just doing bookeeping, it doesn't need the same horsepower as your worker nodes. So you may be wasting a little bit of money here.
  • Trying to configure the cluster to serve outside traffic is painful. For some hosting providers Kubernetes makes it easier, but there is still a lot of manual futzing and it took me quite a bit to get it working.
  • YAML and JSON config file formats are either limited, or annoying, depending on which one you use.

Nice Things

So it's not all vinegar as well, there are a few super nice things they got right:

  • Intra-cluster networking. Want to have a reverse proxy rewrite URLs to send requests to the proper backend? Easy. Just create a Kubernetes service that points to your pods, and then have your reverse proxy talk to the hostname for the service. Doing the same thing on raw docker requires evaluating environment variables to dynamically at container startup to modify your configuration.
  • Did I mention no host side configuration? It took one command to bring up the cluster, then it was just a matter of getting the service/pod config files setup to deploy my containers.
  • GKE comes with an automatic private Docker image repository, which saves you from having to configure it yourself.
  • Kubernets makes it easy for you to distribute secrets (like auth keys) to your worker nodes in a safe an secure manner.


Kubernetes is going to be awesome. Right now, there are a bunch of rough corners and anything you do will probalby give you a splinter. But it still works, there's just a bit of a learning curve.

Borg, Borglets, and Borgmasters. Oh My!

I've been messing with EC2 and cloud computing for some time now. And it's always frustrated me how hard it was to deploy software, versus how easy it was for me to deploy software at work.

Prior to containers like Docker, bring up a new machine was a chore. If you took good notes that last time you did it, that helped (I never did). But chances are, even with your notes, some stuff had changed since the last time you did it.

Enter Docker and containers. I've talked (briefly) about Docker before, but as a refresher you build a container, and can simply deploy it on any machine you like (among many other important things). I say simply, and at first I truely believed it. But once I started running it, I found quite a few gnarly points to deal with. Like the fact that when your container crashes, it's up to you to restart it. Or if you want logs, feel free to configure it on the host. So much for easily moving containers from host to host, you still need to take notes on how your machine was originally setup.

At Google, this has been a solved problem for quite some time. Our production infrastructure handles almost all this for you. You build an image that contains the binaries you need, list the dependent images you need, fill out some constraints on how you want it to run, and hand it to the Borgmaster and it takes care of it for you. Now, I wouldn't call this system user friendly. The constraint language is SUPER complicated (and also SUPER flexible), but at least it's documented. Personally, I came up with a config file back in 2009 and have been using it as the basis for each new service I need to bring up.

I got really excited when I attended OScon last year and learned about Kubernetes (from Google). Being familiar with Borg, I knew immediately what they were talking about. They were re-building our internal Borg systems to run on EC2 (And of course, Google Compute Engine).

And now Google has gone ahead and made a research paper available describing it. (like other core Google tech like Bigtable and Spanner). If you run an internet stack somewhere, you should go read it. If you find research papers to be a little slow going and want a quick read, check out this blog post from the Kubernetes team describing how things learned when developing Borg influenced design decisions in Kubernetes.

One of the reasons I'm uncomfortably excited about Kubernetes is that I trust the people that brought me Borg, to get their second system right. I don't have that same trust from the Docker team, as I've seen them punt on issue that I felt they should also be solving, and in general not always go in the direction I felt was right.

So let's be honest. This isn't all for the greater good, Google expeects to turn a profit on this. Google offers Google Container Engine, which is a hosted kubernetes solution, just like Borg. I'm really looking forward to this system getting more mature, so I can simplify the amount of sysadmin work I need to do to run the small internet services I run.

Code First, or Design First

This post is pretty much a response to this blog post from 18F. 18F Discussion: Should Project Teams Code First Or Design First?. While it pretty much stands alone as well, you might want to go read the first article to put thing in context.

Let me put my old guy hat on for a bit. I’ve been doing this a long time and the answer has always been design first. I think that developers today get too tied up with the power of a strong framework. Yes, your framework and your skills are good enough that you can do this in a few hours and have something passable for the client to view. The question is really, should you? And I think (rather strongly) the answer is no.

The underlying problem is that writing code is slow. Drawing pictures on paper (or whiteboard) is fast. If I sketch something out and it’s bad, or has bad parts, I just throw the paper out and start over again. Maybe incorporate good things from the initial draft and leave out the bad things.

I’ve always instilled in my teams the power of the design review. Once you’re done with a design, sit down and talk to the rest of the team about it. They’ll hopefully help you find things that are bad, things where their experience may give you feedback you wouldn’t have seen yourself, etc. And the best part is, the only thing you’re invested in is the documentation. It’s easy to move words around in a document, or move blocks in a diagram.

With code that even works a little bit, there is an impetus. You’ve invested time in getting it to work this well, you’re going to be hesitant to change it, which can keep you closed minded when discussing alternative solutions. Or even worse, the customer sees your demo and latches on to it, thinking it’s only a trivial amount of work to get it to launch.

When I worked at Hillcrest Labs, we had the prototype that wouldn’t die. It was all fake stuff that would only ever run on a PC (we were targeting an embedded platform). It took a lot of work to convince our management that we needed to concentrate on only working on the real product and remove everyone from working on the prototype. And eventually we did, but it was an uphill battle.

We can also talk about the quality of prototype code versus production code. The thinking is “This is just a prototype, I don’t need to handle this error here.” Which means that when it becomes time to do it for real, you either go through all your existing (crappy) code and try to patch it up to production quality, or you start over and all your work was for naught.

So, while an interesting article, I don’t really think there is really much of a discussion. Coding first just has too many risks.

Dynamic Queries to your App Engine Data Store

I'm using AppEngine for a project at work. Exactly what it does it not important, but at some point it stores an Event entry into the AppEngine datastore for each thing that we do. I put this is so we could easily show the lsat 5 events on a webpage, and also because it felt like a good idea at the time.

This tool has been running for about 6 months now, we've got a bunch of events stored. I need to look at this data to see how long the events are taking, so I can have an idea of what a good event timeout value would be. A simple approach would be to write some python code in my application to render a custom admin page that could display this information for me. This approach is fraught with issues:

  • More code to write, which means more passes through the write, deploy, test, debug cycle. (The debug datastore doesn't have enough interesting information to do this locally.)
  • I've got over 7000 entries in the datastore. To do this serially is going to be kinda slow. I'm not positive I can do it before the AppEngine request deadline kicks in. Or if I can beat the deadline now, I won't be able to in the future.

So I started looking around and came up with another approach. AppEngine supports a datastore backup feature, and BigQuery can use that data as an input. Bingo. I can now treat my datastore as if it were SQL and write dynamic queries to explore the data that way.

The Nitty Gritty

So there are a few things to setup. First, you need a Google Cloud Storage bucket to store your stuff in, and you need to give your application access to write to it. If you are lucky enough to be using the default bucket associated with your app, then you have nothing to do but look up your bucket name (it's in the cloud console for your app engine app).

Setup Cloud Storage

Start by going into your AppEngine settings page and go to "Application Settings". Under there we want to find and note your "Service Account Name". This is the Google user your app runs as. Remember it, we'll need it for the next step.

Go into the cloud console for your specific app. If you didn't create one explicity, Google created one for you. The application id for your app should appear somewhere on this page. Go into it and create a new bucket, and then give the Service Account from above write acccess to this bucket. (Again, if you're using the default bucket for your app, this already setup for you).

Do the backup

Find the "Datastore Admin" tab in your AppEngine settings page, select the entity type to backup, and start your backup. Make sure to point it to your Cloud Storage bucket when asked.

Setup BigQuery

Go back to the cloud console for your specific app and scroll down to find BigQuery. You'll now need to create a datastore (call it whatever you want, I called mine test) and then create a table inside that datastore (I also called my table test), and point it to the file inside your bigstore bucket. Mine looked something like:


Now you're ready to start examining your data.

Use BigQuery

BigQuery's query language is very similar (but not exactly) SQL.

A simple example, how many events did I have?


Told me I had over 7000 entries. But I needed something more interesting. My events can be grouped around a key, and I'm looking for information on how long they took. The query I used was this one:

  AVG((END-start)/1000/1000/60) AS avg_duration,  
  MAX((END-start)/1000/1000/60) AS max_duration,
  MIN((END-start)/1000/1000/60) AS min_duration,
  COUNT(*) AS cnt
  status == "complete"
  avg_duration DESC

Which basically says: For each key, show me the min, avg, and max duration of "complete" events, ordered by average duration. For ease of the human eye, I did all my durations in minutes, because that worked for my data.


7000 entries? I've forgotten how to count that low. -- BigQuery

So yes, BigQuery is overkill for this dataset. I could have just whipped up a python script, or dumped it into MySQL and done it that way. But the advantage of this way is that all the bits are already setup. I just needed to hook up the outputs to the inputs and I was ready. I didn't even need to download anything to my local machine, it all took place "in the cloud".

But I view this setup as just the start of something. Yes, I solved my immediate question, but what else could I do? How about setting up daily backups, and then scripting up a daily import of those into BigQuery. Then whenever I need to do a quick query on something, the data is already there.

Or going even further, how about some data visualizations based on BigQuery? If I have a snapshot for every day, I could graph some things like events per day, average durations per day, errors per day, etc.

So yes, it's still a bit of work to get all this glued together such that we can work on it. In my mind, there is still lots of room for improvement, but at least in this case Google delivers tools that work with other Google products in a meaningful fashion. That's not always the case.

I Read These 10 Books, and You'll Never Believe What Happened Next!

What happened next is that I got on with my life. But it got you to click through, didn't it? Anyway...

Margaret and I were discussing one night our 10 favorite books of all time. 5 was pretty easy to get to off the top of my head, but filling in the last 5 was a bit harder. What helped fill it in was going to my bookshelf and taking a look. I tend to keep books I like.

Rather than just list these out on Facebook, I decided that it would be more interesting to actually write a little about each book. So here goes.

The Books

The Hitchhiker's Guide to the Galaxy (by Douglas Adams)

I've lost track of how many times I've read this series cover to cover. I've got a well worn copy of first four in the series and a standalone copy of Mostly Harmless. Adams has such a unique style of story telling. Kinda manic and all over the place, with a lot of (typically British) humor of the absurd. It's never really overtly comic, and yet still has a number of laugh-out-loud moments.

Hey, you sass that hoopy Bill Napier? There's a frood who really knows where his towel is.

The name of this blog actually comes from this book.

Lord of the Rings (by JRR Tolkein)

T'was in the darkest depths of Mordor, I met a girl so fair. / But Gollum, and the evil one crept up and slipped away with her, her, her....yeah.

I was introduced to LotR rather late compared to most other geeks. I didn't read it the first time until High School. And the only reason I really leared of it was because I was listening to a lot of Led Zeppelin at the time, and a few of their songs make reference to the series (Misty Mountain Hop).

LotR was my first exposure to what I call "Epic Fantasy". There is a huge, detailed world and the reader is thurst into it with no background and has to figure things out on the fly, much like the characters in the book. I love how detailed Tolkein made Middle Earth, where even things like the phase of the moon it properly described.

Snowcrash (by Neal Stephenson)

Didn't find out about this novel or Stephenson until college, again kinda late to the game. The opening of the book is really what grabbed me. Name another book that starts off with a pizza delivery and the mob. Where does he come up with this stuff?

Snowcrash was published in 1992 and most of what Stephenson talked about was totally fantastic, definite Science Fiction. I reread the book sometime later (2004?) and was gobsmacked to see how many of the things Stephenson made up were now actual products. Google Earth. Second Life. I bet startup companies today are going back to this book today to find out their next idea, I'm sure it's in there.

Ender's Game (by Orson Scott Card)

Remember, the enemy's gate is down.

Again, discovered this book in college (same guy who recommended Snowcrash). This was back before Card starting running off his mouth about gay people, so it was still OK to read him.

A lot of Ender's game could be described as pretty typical Young Adult SF, like Heinlein "Juveniles". Exceptional kid, picked to save the world, smarter than the adults, has to overcome the bullies jealous of him. We all wanted to be Ender. All pretty typical YA coming of age store. If it weren't for the ending, this book never would have made the list. I won't spoil it for you here (even though the movie spoiled it in the trailer...), but there's quite a twist at the end that catapults a good story into a great one.

Sadly, pretty much everything else Card has written is dreck.

Name of the Wind (by Patrick Rothfuss)

Words are pale shadows of forgotten names. As names have power, words have power. Words can light fires in the minds of men. Words can wring tears from the hardest hearts.

Of all things, Margaret's randomly bought this one for me. She has a history of picking winners for me, and this one is no exception. Another epic fantasy, this time with a believable magic system underpinning the story. If you've read the Dresden Files, you'll be familiar the magic system as Rothfuss has borrowed heavily from it (he's admits he's a fan).

But the store is so much more than that. Starts off as a pretty standard YA coming of age story, like a more adult version of Harry Potter (including the school!). Except Kvothe get's kicked out of school and then we're off on an adventure.

Have no idea how this one's going to end. He's promised a triolgy and has only written two of them. There's quite a bit of story to cover in a single book....

Dune (by Frank Herbert)

My own name is a killing word

I took a course in Science Fiction as a Junior at Penn. In addition to lecture (which basically covered a lot of this history), we had to read two books each week and be able to write about them for recitation.

This weeks theme was "Epic Novels" (I'm sensing a theme). We had to read Le Guin's The Dispossessed and Herbert's Dune. I stared with Le Guin's book and it took most of the week for me to get through it. Even though it won basically every SF award possible (Nebula, Hugo, Locus), to this day I have no idea what it was about.

I was in a bind. I had recitation in under 24 hours and all 500+ pages of Dune to read. My plan was to get through enough of the book to be able to talk about it, and finish it after recitation. What ended up happening was that I stayed up all night to finish it. I couldn't put it down. Again, another Epic story of a fantastic world and of interesting people. None of the movies for Dune do the book justice.

The Robots of Dawn (by Issac Asimov)

The robot had no feelings, only positronic surges that mimicked those feelings. (And perhaps human beings had no feelings, only neuronic surges that were interpreted as feelings.)

It was summer, probalby during high school. Church Youth Group trip to Ocean City, NJ. I needed something to read, so I picked this up at the bookstore. I proceeded to get very sunburnt on the back of my legs while reading it on the beach.

This book affected me so much that I still, 20 years later, remember where I read it. It was probalby the first hard science fiction I've read (at least that made this list), and some of the idea still stick with me. Of course, Asimov's 3 laws of robotics (and the 0th law). But also the "people mover" idea, where there are treads moving at different rates, and you can pay more to ride a faster tread. Or pay a premium and get a seat.

So this got me hooked on Asmiov. I went back and read the earlier books in the series, but this was by far the best. This one is where the ideas were more mature, more distilled. The earlier books still seemed a touch simplistic.

Why not Foundation? To be honest, it's probably the better series. But I couldn't tell you where I was when I read it the first time (The second time was spring break during college for class).

Hackers (by Steven Levy)

Systems are organic, living creations: if people stop working on them and improving them, they die.

In college I got really hooked on the history of computing. Still am. Soul of a New Machine (Data General). Revolution in The Valley (Apple). In The Plex (Google). Some book on a company who failed (Sorry, don't remember the name). The book on the history of the iPod.

But this book is by Stevn Levy (just like the iPod book and the Google book). It's one of his earlier works, and I actually like it better than some of this later works. Levy has a tendancy to write his books as a series of magazine articles. In Hackers, it works because he split the book into a series of separate stories. His other books it's more annoying.

And it talks about all the stuff that happened when I was a kid, too young to participate. The Altair. The original Apple. Woz and Jobs back when they were the dynamic duo. Homebrew Computing Club. The good old days.

On a Pale Horse (by Piers Anthony)

What kind of fool had he been, to throw away romance untried?

I read a lot of Piers Antohny as a kid. And by a lot, I mean pretty much everything he wrote. I loved his work. Eventually I realized that most of most famous books (The Xanth Series) were super formulaic and started to get boring.

On the other hand, his Incarnations of Immortality series Bio of a Space Tyrant were pretty good, at least in the mind of a 12 year old boy.

Imagine that Death is job that you can work for eternity. He (and his friends Time, War, Fate, and Nature) are in a constant battle against Satan. It blew my mind and I really enjoyed reading it.

Recently I started re-reading the series. It's been nearly 30 years, and I still remember quite a bit of it. And I'm finding that it doesn't hold up that well. Not that's its dated, but more that I'm older now and things that seemed profound to a 12 year old have been proven false via experience. Plus Anthony seems to be a touch of a woman-hater, which can make it hard to read.

Phantom Tollbooth (by Norton Juster)

Ok, this is weird one. I read it in junior high. I don't really remember why it was so impactful on me, but anytime a book for a 10 years old comes up, this is the one that I point out. Looking forward to Joshua being old enough to read it so I can get him a copy.

Didn't Make The List

There were a few books that I like, but felt just got bumped off the list. I'll list them here in case your interested.

  • Old Mans Way by John Scalzi. Very good book, really enjoyed reading it and the rest of the series, but nothing from it has really stuck with me.
  • Game of Thrones by George RR Martin. Only started reading this due to the HBO series, trying to stay ahead of spoilers. It's very very good, but I don't ever see myself re-reading it. The good points are too few and far between, especially in the later books.

So, what are some of your favorite books?