Category: Geek

Netlify + Cloudflare = Crazy Delicious

January 29, 2020 » Geek

At this years NEJS Conf, Netlify’s Phil Hawksworth gave a highly entertaining talk about mostly static content as a new normal path. His demo was a great little app that showed the possibilities of mixing static rendering and a dash of just in time functions as a service.

Previously I had not been a big fan of the JAMstack concept, it hadn’t clicked for me and seemed very single page app oriented. This demo piqued my interest, so I decided to move cultofthepartyparrot.com over to try out it’s automated deployment systems.

It’s magic.

Seriously, it’s remarkable how well it worked. CotPP has a custom build script and a lot of weird dependencies, and it all built with minimal intervention.

I set the GitHub repo as its source, pointed the build command at my custom script, and let it rip. The first build failed because I messed up the build command name. The second build had a live version for me to view in 50 seconds. That’s pretty great for installing all the tools, generating and deploying it. I was hooked.

I promptly pointed over the domain, got SSL issued and considered it done. And did I mention it creates deploys for all PR’s? I could finally preview new parrots in situ before a merge. Pretty amazing for a free product.

Uh-oh.

The Cult of the Party Parrot has been hosted on a shared Dreamhost server since its creation. I have Google Analytics on there, so I had some idea of the amount of traffic that it received, but never paid much attention. Turns out, it uses a lot of bandwidth.

Within five days I had used 50GB of traffic on Netlify. That means I’d be paying ~$60 a month for hosting, which isn’t viable for me, as CotPP doesn’t have ads or anything. I needed a way to either keep using Netlify, or recreate the Netlify experience (with the PR deploys) in some other toolkit.

My first instinct was to just go back to Dreamhost and figure out the automatic deploys using an existing tool like GCP CloudBuild. But then, the ever reliable and always clever Ben Stevinson suggested that I put Cloudflare in front of it, and speed it up in the process.

That sounded like a good idea to me, if I could get Cloudflare to catch the bulk of the bandwidth, then the 100GB cap of the Netlify free plan should be plenty to host the PR deploys, and I can have the best of both worlds, with the least amount of effort.

Putting Cloudflare in front of Netlify works just fine. I transferred DNS to Cloudflare, and then had it CNAME flatten to the Netlify origin. TLS was easy, and setting up the Cloudflare origin certificate on Netlify was simple too. Finally, I added a page rule that tells the Cloudflare edge to cache everything for a month. Bandwidth problem solved.

But, there was one last issue. Every time Netlify did an automatic deploy for me after I closed a pull request, I would have to manually go in and flush the cache on Cloudflare. That’s no good. The solution was to connect a Netlify deploy notification webhook to a GCP cloud function which clears the cache via the Cloudflare API.

Netlify deploy webhook configuration modal.

The documentation on the Netlify webhook is a little light, so I ran a few deploys and just printed out the contents to find the keys I need. Here’s an abridged example output of what I get in the webhook body.

All I really cared about there was branch, and the ID could be useful for tracing deploys to flushes if something went awry. So with that in hand I started putting together my function. The struct for unpacking the webhook is pretty small, and there’s nothing novel going on there.

You may note that I used a io.TeeReader there to duplicate the request body reader into a buffer. This is used later when validating the JWT that Netlify sends, more on that later.

Once unpacked, we can check that this update is for the master branch before we proceed. If we flushed on every PR deploy it would be a waste of effort, so we only want to proceed for a merge into master.

Now we want to verify that this request really did originate from Netlify. Now, for this use case it probably doesn’t matter that much, who is going to take the time to figure out the details of my cloud function and launch a purging spree to run me out of Netlify bandwidth? But it’s easy to implement, and we can learn something along the way, so why not!

A Brief Aside About JWT

A JWT is a “JSON Web Token”. It’s a three part string consisting of a header, a payload and a signature. The header and payload are JSON that is base64 encoded, and the signature is an HMAC of the other two sections, again base64 encoded. The header tells you things about the token itself, such as the algorithm used to create the signature. The payload is arbitrary data, though there are standardized fields, or “claims” in JWT parlance. You can learn more about it at jwt.io, but that should be enough to get us through this

On to the validation!

Since JWT is a well known standard, we have several packages to pick from for validating them. I chose github.com/gbrlsnchs/jwt, which had the API I liked best.

First we need to define the payload we expect from Netlify. Their payload is very simple with just two claims, as their docs say:

We include the following fields in the signature’s data section:

  • iss: always sent with value netlify, identifying the source of the request
  • sha256: the hexadecimal representation of the generated payload’s SHA256

Next we send the body of the request, the JWT, and an empty struct to jwt.Verify.

The variable hs here is an instance of an HMAC hashing function, specifically a jwt.HS256, since the Netlify hook always uses that algorithm to sign it’s JWTs. That is initialized elsewhere using a secret pulled from the environment.

Once the JWT is validated and the payload extracted from it, we hash the contents of the request body with SHA256. Remember that io.TeeReader? This is what we stashed the body for. We compare the hash we derived from the one in the payload to ensure the body was not tampered with in-flight.

Once everything checks out, we make the request to Cloudflare to purge the whole zone. This is an API method available on all Cloudflare plans, Purge All Files

Then we’re done! We just have to convey the status of the API call as our status to bring it all together.

122.5GB cached bandwidth in a month, 2.37GB uncached.

Overall I’m quite happy with this solution. Perhaps it’s a bit over engineered, but it’s saving a ton of money I don’t have to burn on CotPP, and I don’t have to move it back to Dreamhost either.

You can get the full code for this on Github in the CotPP repo on Github.

Easy visual identification of git short sha’s.

January 9, 2019 » Geek

For a recent pet project at work I had to display a bunch of git short shas. Most of the time, these shas should match each other, and it is important to be able to quickly glance at them and evaluate if any of them are not the same.

Sure, you could count on your eyes to just notice the characters don’t match, but short shas are drawing from a limited alphabet (16 characters) and we would only be adding more shas to the listing over time, so noticing one abberant item would get harder over time.

The solution I landed on was to drop the final character of the sha and use that as a hex color string. While this is imperfect, it works remarkably well.

To make the sha readable over the color I needed to find a contrasting color. To keep it simple, I took the average of the individual channels, and if it was over 128 I used black. Under 128 I used white.

Here’s a little snippet of the JavaScript I wrote for this:

This could be improved by using luminosity measures designed for eyeballs instead of a rough mean of the colors, I could constrain the colorspace for shas to throw out colors that are problematic for color blindness, etc. I’ll leav e that as an exercise for the reader.

Tags: ,

Using environment secrets as build arguments in Google Cloud Build

December 27, 2018 » Geek

Google Cloud Build is a pretty nice tool for building your docker images continually, and cloud-build-local is pretty great for working on your images in dev. All around, a nice piece of kit to have in a Kubernetes shop.

The docs are pretty good, but one thing that I’ve recently dealt with did not show up in my searching; how to use an environment secret as a build argument to Docker. So here’s how I found to do it.

First, we will follow the encrypted secrets guide to get a secret wrapped up by KMS.

Next, we will create a super simple Dockerfile to show how it is used.

Last, we set up the cloudbuild.yaml. In the documentation demo files they use a shell entrypoint to access the environment variable.

However, it would be nicer to not have to stringify our whole Docker build command.

Luckily, using --build-arg without a value falls through to the environment variable of the same name.

So, we can just use it directly:

Testing locally, it happily runs:

It is worth noting that using build args for secrets is not recommended. Anyone with the image can see what the argument passed in was.

Docker 18.09, added build secrets for a better solution, but GCB is still running Docker 17.12, so we will have to wait for that update.

A gist of the code is available at: https://gist.github.com/jmhobbs/a572b47048eb42803bcb2102ac57a8df

Party Gopher!

June 13, 2018 » Geek

The Go slack has a cute little dancing Gopher that appears to have come from Egon Elbre. I love it!

Dancing Gopher

This little dancing Gopher made me think of Party Parrot, so I wanted to parrot-ize him. Normally I might just open up Gimp and start editing, but this is the Go Gopher, we can do better than that!

My plan was to use Go’s image packages to edit each frame and replace the blue with the correct parrot color for that frame by walking over the pixels in each frame.

Once I got into the package docs however, I realized that since gif’s are paletted, I can just tweak the palette on each frame and be done. Much simpler. Let’s get into then, shall we?

Colors!

First things first, I needed to declare the party parrot frame colors, and the light and dark blue that the dancing gopher uses. I grabbed the blues with Sip and I already had the parrot colors on hand. Sure, I could precompute these and declare, but let’s keep it interesting.

Note that I have a DarkParrotColors slice as well, this is for the corresponding dark blue replacements. I generate these with darken which I’ll show in a moment.

Also notable is the hexToColor which just unpacks an HTML hex RGB representation into a color.Color.

Here is the darken function, pretty simple.

Now I need to pull in the gif and decode it, all very boilerplate.

After that, I iterate over the frames and edit the palettes.

Lastly, more boilerplate to write it out to disk.

Party Gopher

You can grab the code on Github, and thanks again to Egon Elbre for the excellent original gif!

A New GPG Key

May 30, 2018 » Geek

It’s been 12 years since I created my first GPG key and 11 since I’ve created the one I actually use. That is far too long, so I decided to create a new pair and deprecate the old. In 2013 I started this process, but I didn’t follow through and I’ve since lost access to those keys. I know where they are, but the machine died so I need to hook up it’s HDD and pull the keys out.

Regardless, it is time for new ones, and I did some reading to get a real plan for this. I would generate a new, strong key offline, with a subkey for each capability. The subkeys would go onto a smart card, in my case a Yubikey 4. The primary key material would go to offline backup to keep it safe.

Disclaimer

Nothing in this post is new or novel, but rather collected from many other posts. I’ve tried to link to any relevant posts below each section, and I encourage you to read these sources. Any mistakes I’ve made I would be glad if you send me an email (GPG encrypted of course ;) to point it out.

Yubikey Configuration

After I ordered my Yubikey, I had to configure it. The Yubikey docs expect a fair amount of knowledge before you start, but the steps are pretty simple when you understand it. Basically, it boils down to:

  1. Change the Admin PIN
  2. Change the PIN
  3. Set a Reset Code
  4. Fill in optional metadata

Plug in your card and proceed as follows:

Links

Generating Keys

Next, I created my keys. Be sure you set up a clean environment for this, ideally a random directory in /tmp, better still on a ramfs of an offline, live CD machine. But that’s a bit drastic for my use case.

You’ll want a good base config file in there too.

With the directory in place, I can create a primary key, option 4. 4096-bits is as strong as GPG allows right now, and I set it not to expire because I will be keeping offline and it should be ok to revoke manually if needed.

Now it’s time to create subkeys. There are four capabilities that a PGP key can have.

C is for Certify

Your primary key will have the capability of Certification. Certify is essentially the ability to sign other keys. A key with Certify can be “parent” to subkeys, create new subkeys, and edit existing ones. You also need this capability to sign another users public key.

S is for Sign

A key with the Sign capability can sign files and messages, allowing others to verify their integrity.

E is for Encrypt

A key with the Encrypt capability is used for encrypting files. Simple.

A is for Authenticate

An Authentication key is generally used for SSH authentication.


Generating the subkeys is a bit tedious, but so it goes.

The authentication key requires E X P E R T M O D E. Git gud.

That’s it! We’re in business.

Links
  1. https://spin.atomicobject.com/2013/11/24/secure-gpg-keys-guide/
  2. https://www.linode.com/docs/security/authentication/gpg-key-for-ssh-authentication/
  3. https://gist.github.com/abeluck/3383449
  4. https://alexcabal.com/creating-the-perfect-gpg-keypair/
  5. https://gist.github.com/graffen/37eaa2332ee7e584bfda

Backups

Before we do anything else, we need to back that thang up.

I’m choosing two methods: backup to a USB key that will live in a fire safe (who has a safety deposit box these days?), and a printed backup in case the USB key fails. Ideally these two articles would not be co-located.

First we export the keys and move them to the USB stick. The export-secret-subkeys output is less important than the export-secret-key output as it doesn’t contain a viable certification key, but would be useful as a “middle tier” of backup that wouldn’t expose your primary key to risk.

Now, we could take these ascii armored keys and just print them, but that’s a lot of bytes to pray for OCR to recognize. Instead, we can use a piece of software called Paperkey which strips out everything but the most secret parts of the key and gives you something much shorter to type in.

Still not fun to type it all in, but it’s better and this is a last ditch sort of thing anyway.

Recovery

Backups you don’t test aren’t backups, they are hopes and dreams. So let’s try recovering from our paperkey output!

Links
  1. http://www.jabberwocky.com/software/paperkey/

The certificate revoke you, secret key!

While not required, we can generate a revocation certificate while we still have the primary key on this machine.

Throw that onto your backup drive too while you’re at it.

Links
  1. https://www.hackdiary.com/2004/01/18/revoking-a-gpg-key/

Sign!

Ok. Everything is generated, we have a good backup, we are ready to transition. To indicate that this key is your new key, you can sign it with your old one, then send it up to the keyservers in the sky (if you’re into that)

Links
  1. https://www.apache.org/dev/key-transition.html

To The Smart Card Robin!

I'm Batman

Moving the keys onto a smart card helps protect them. They won’t exist on your filesystem anymore, only on the card. That means they can’t be read out and stolen by a malicious process, but you can still use them by providing your smart card pin and key password.

Keep in mind, this is a one way trip. Make sure your backups are really, truly in place. We want to move our Signing, Encryption and Authentication keys onto the card. The Certification key we will only store offline, as mentioned before.

HOYB

Hold onto your butts.

This is it. The big moment. Take out that smart card, secure your backups, and let’s delete our primary key material.

Now you can have gpg create key stubs for all the keys on your smart card.

Now when we list keys, we see that our primary key has a # next to it, showing we don’t have access to that secret key. The subkeys have a > next to them showing they are stubs for the keys on the smart card. Success!

Fin!

That’s it. There is, of course, more to do, like setting up git signing, SSH access, etc. But the new keypair is created, and it’s on the Yubikey, so that’s all for now.


Update: Git Signing

Turns out git signing is a cinch. Just throw a couple items into your git config and it’s automatic and transparent.

user.signingKey

Tells git which key to use for signing, unset it just uses the default key.

commit.gpgSign

Makes it sign all commits by default, instead of passing -S to every git commit.

log.showSignature

By default, git won’t show you if a commit is GPG signed. You can see it with gpg log --show-signature, or you can set it as default with this config option.

It makes signed commits much chunkier, so be aware of the reduced screen real estate.

merge.verifySignatures

This is the only one I am not setting by default. If you have it enabled, all merges that include unsigned commits will be rejected. This really only works if everyone in your organization is signing all their commits.

Links
  1. https://git-scm.com/book/en/v2/Git-Tools-Signing-Your-Work
  2. https://git-scm.com/docs/git-config

Update: One-Touch Actions

By default, with the smart card in, GPG will happily sign and decrypt things after you enter your PIN the first time, with no further interaction from you. The Yubikey offers a mode where these actions require a touch on the key to complete, which I like because it makes the action more explicit without requiring me to remove the key between operations.

To enable this, you need a special script, yubitouch.sh. To make it work with my GPG Tools install, I had to hard code the path to gpg-connect-agent (/usr/local/MacGPG2/bin/gpg-connect-agent) and my admin PIN, since pinentry wasn’t working and I didn’t want it in my bash history.

Now, when GPG needs to sign something, my Yubikey flashes at me until I touch it and give my permission. Neat.

Links
  1. https://developers.yubico.com/PGP/Card_edit.html
  2. https://github.com/a-dma/yubitouch
Tags: , ,