Tag: Google Cloud Platform

Netlify + Cloudflare = Crazy Delicious

January 29, 2020 » Geek

At this years NEJS Conf, Netlify’s Phil Hawksworth gave a highly entertaining talk about mostly static content as a new normal path. His demo was a great little app that showed the possibilities of mixing static rendering and a dash of just in time functions as a service.

Previously I had not been a big fan of the JAMstack concept, it hadn’t clicked for me and seemed very single page app oriented. This demo piqued my interest, so I decided to move cultofthepartyparrot.com over to try out it’s automated deployment systems.

It’s magic.

Seriously, it’s remarkable how well it worked. CotPP has a custom build script and a lot of weird dependencies, and it all built with minimal intervention.

I set the GitHub repo as its source, pointed the build command at my custom script, and let it rip. The first build failed because I messed up the build command name. The second build had a live version for me to view in 50 seconds. That’s pretty great for installing all the tools, generating and deploying it. I was hooked.

I promptly pointed over the domain, got SSL issued and considered it done. And did I mention it creates deploys for all PR’s? I could finally preview new parrots in situ before a merge. Pretty amazing for a free product.


The Cult of the Party Parrot has been hosted on a shared Dreamhost server since its creation. I have Google Analytics on there, so I had some idea of the amount of traffic that it received, but never paid much attention. Turns out, it uses a lot of bandwidth.

Within five days I had used 50GB of traffic on Netlify. That means I’d be paying ~$60 a month for hosting, which isn’t viable for me, as CotPP doesn’t have ads or anything. I needed a way to either keep using Netlify, or recreate the Netlify experience (with the PR deploys) in some other toolkit.

My first instinct was to just go back to Dreamhost and figure out the automatic deploys using an existing tool like GCP CloudBuild. But then, the ever reliable and always clever Ben Stevinson suggested that I put Cloudflare in front of it, and speed it up in the process.

That sounded like a good idea to me, if I could get Cloudflare to catch the bulk of the bandwidth, then the 100GB cap of the Netlify free plan should be plenty to host the PR deploys, and I can have the best of both worlds, with the least amount of effort.

Putting Cloudflare in front of Netlify works just fine. I transferred DNS to Cloudflare, and then had it CNAME flatten to the Netlify origin. TLS was easy, and setting up the Cloudflare origin certificate on Netlify was simple too. Finally, I added a page rule that tells the Cloudflare edge to cache everything for a month. Bandwidth problem solved.

But, there was one last issue. Every time Netlify did an automatic deploy for me after I closed a pull request, I would have to manually go in and flush the cache on Cloudflare. That’s no good. The solution was to connect a Netlify deploy notification webhook to a GCP cloud function which clears the cache via the Cloudflare API.

Netlify deploy webhook configuration modal.

The documentation on the Netlify webhook is a little light, so I ran a few deploys and just printed out the contents to find the keys I need. Here’s an abridged example output of what I get in the webhook body.

    "admin_url": "https://app.netlify.com/sites/cultofthepartyparrot",
    "available_functions": [],
    "branch": "master",
    "build_id": "00000000e8ffde017674b0b2",
    "commit_ref": null,
    "commit_url": null,
    "committer": null,
    "context": "production",
    "created_at": "2019-08-26T21:04:48.294Z",
    "updated_at": "2019-08-26T21:05:31.385Z",
    "url": "https://cultofthepartyparrot.com",
    "user_id": "000000011111112222222333"

All I really cared about there was branch, and the ID could be useful for tracing deploys to flushes if something went awry. So with that in hand I started putting together my function. The struct for unpacking the webhook is pretty small, and there’s nothing novel going on there.

type netlifyWebhook struct {
	ID     string `json:"id"`
	Branch string `json:"branch"`

func PurgeCloudFlare(w http.ResponseWriter, r *http.Request) {
	var bodyBuf bytes.Buffer
	tee := io.TeeReader(r.Body, &bodyBuf)
	defer r.Body.Close()

	dec := json.NewDecoder(tee)

	var wh netlifyWebhook
	err := dec.Decode(&wh)
	if err != nil {
		log.Println("error decoding webhook body:", err)
		http.Error(w, err.Error(), http.StatusInternalServerError)

You may note that I used a io.TeeReader there to duplicate the request body reader into a buffer. This is used later when validating the JWT that Netlify sends, more on that later.

Once unpacked, we can check that this update is for the master branch before we proceed. If we flushed on every PR deploy it would be a waste of effort, so we only want to proceed for a merge into master.

	if wh.Branch != "master" {
		w.Write([]byte("Ok. Thanks."))

Now we want to verify that this request really did originate from Netlify. Now, for this use case it probably doesn’t matter that much, who is going to take the time to figure out the details of my cloud function and launch a purging spree to run me out of Netlify bandwidth? But it’s easy to implement, and we can learn something along the way, so why not!

	jwt := r.Header.Get("X-Webhook-Signature")
	if jwt == "" {
		http.Error(w, "Forbidden", http.StatusForbidden)

	if !verifyRequest([]byte(jwt), bodyBuf.Bytes()) {
		http.Error(w, "Forbidden", http.StatusForbidden)

A Brief Aside About JWT

A JWT is a “JSON Web Token”. It’s a three part string consisting of a header, a payload and a signature. The header and payload are JSON that is base64 encoded, and the signature is an HMAC of the other two sections, again base64 encoded. The header tells you things about the token itself, such as the algorithm used to create the signature. The payload is arbitrary data, though there are standardized fields, or “claims” in JWT parlance. You can learn more about it at jwt.io, but that should be enough to get us through this

On to the validation!

Since JWT is a well known standard, we have several packages to pick from for validating them. I chose github.com/gbrlsnchs/jwt, which had the API I liked best.

First we need to define the payload we expect from Netlify. Their payload is very simple with just two claims, as their docs say:

We include the following fields in the signature’s data section:

  • iss: always sent with value netlify, identifying the source of the request
  • sha256: the hexadecimal representation of the generated payload’s SHA256

type netlifyPayload struct {
	ISS    string `json:"iss"`
	Sha256 string `json:"sha256"`

Next we send the body of the request, the JWT, and an empty struct to jwt.Verify.

func verifyRequest(token []byte, body []byte) bool {
	var pl netlifyPayload
	_, err := jwt.Verify(token, hs, &pl, jwt.ValidateHeader)
	if err != nil {
		return false

The variable hs here is an instance of an HMAC hashing function, specifically a jwt.HS256, since the Netlify hook always uses that algorithm to sign it’s JWTs. That is initialized elsewhere using a secret pulled from the environment.

func init() {
	hs = jwt.NewHS256([]byte(os.Getenv("JWT_SECRET")))

Once the JWT is validated and the payload extracted from it, we hash the contents of the request body with SHA256. Remember that io.TeeReader? This is what we stashed the body for. We compare the hash we derived from the one in the payload to ensure the body was not tampered with in-flight.

	h := sha256.New()

	return pl.Sha256 == fmt.Sprintf("%x", h.Sum(nil))

Once everything checks out, we make the request to Cloudflare to purge the whole zone. This is an API method available on all Cloudflare plans, Purge All Files

url := fmt.Sprintf("https://api.cloudflare.com/client/v4/zones/%s/purge_cache", cloudFlareZone)
	req, err := http.NewRequest("POST", url, bytes.NewBuffer([]byte(`{"purge_everything":true}`)))
	if err != nil {
		http.Error(w, err.Error(), http.StatusInternalServerError)
	req.Header.Set("Authorization", fmt.Sprintf("Bearer %s", cloudFlareAPIToken))
	req.Header.Set("Content-Type", "application/json")

	resp, err := cloudflareHTTPClient.Do(req)
	if err != nil {
		http.Error(w, err.Error(), http.StatusInternalServerError)

Then we’re done! We just have to convey the status of the API call as our status to bring it all together.


	if resp.StatusCode != http.StatusOK {
		body, _ := ioutil.ReadAll(resp.Body)
		defer resp.Body.Close()

122.5GB cached bandwidth in a month, 2.37GB uncached.

Overall I’m quite happy with this solution. Perhaps it’s a bit over engineered, but it’s saving a ton of money I don’t have to burn on CotPP, and I don’t have to move it back to Dreamhost either.

You can get the full code for this on Github in the CotPP repo on Github.

Using environment secrets as build arguments in Google Cloud Build

December 27, 2018 » Geek

Google Cloud Build is a pretty nice tool for building your docker images continually, and cloud-build-local is pretty great for working on your images in dev. All around, a nice piece of kit to have in a Kubernetes shop.

The docs are pretty good, but one thing that I’ve recently dealt with did not show up in my searching; how to use an environment secret as a build argument to Docker. So here’s how I found to do it.

First, we will follow the encrypted secrets guide to get a secret wrapped up by KMS.

$ # Create a keyring and encryption key
$ gcloud kms keyrings create tinkering --location=global
$ gcloud  kms keys create cloud-build-demo \
  --keyring=tinkering \
  --purpose=encryption \
$ Now encrypt our secret string
$ echo -n "This is the super secret secret." | gcloud kms encrypt \
  --plaintext-file=- \
  --ciphertext-file=- \
  --location=global \
  --keyring=tinkering \
  --key=cloud-build-demo | base64

Next, we will create a super simple Dockerfile to show how it is used.

FROM busybox
RUN echo "::${THE_SECRET}::"

Last, we set up the cloudbuild.yaml. In the documentation demo files they use a shell entrypoint to access the environment variable.

# Note: You need a shell to resolve environment variables with $$
- name: 'gcr.io/cloud-builders/docker'
  entrypoint: 'bash'
  args: ['-c', 'docker login --username=[MY-USER] --password=$$PASSWORD']
  secretEnv: ['PASSWORD']

However, it would be nicer to not have to stringify our whole Docker build command.

Luckily, using --build-arg without a value falls through to the environment variable of the same name.

$ export HTTP_PROXY=
$ docker build --build-arg HTTP_PROXY .

So, we can just use it directly:

- id: docker
  name: 'gcr.io/cloud-builders/docker'
  args: ['build', '--build-arg', 'THE_SECRET', '.']
  secretEnv: ['THE_SECRET']
- kmsKeyName: projects/hobbs-tinkering/locations/global/keyRings/tinkering/cryptoKeys/cloud-build-demo
    THE_SECRET: CiQATajs0GI7M6ZFM68Qu+GbJTfJ/d3tqqLcHz69RY1AaHkzV20SSQDt7E4V65imqbOnq8DvieiaglxjEztxWQCwrr2Mtu+xwT6tko6FHB+NNauyos6X1nnh5x217Cwx5QbX3h0YtjOJ15I4dnHDM+I=

Testing locally, it happily runs:

$ cloud-build-local --dryrun=false .
Using default tag: latest
latest: Pulling from cloud-builders/metadata
Digest: sha256:bcdb85e67ab9719c6441cb80fe9e8badc6d5ab0ab8bc73ee67adc0112233d20c
Status: Image is up to date for gcr.io/cloud-builders/metadata:latest
2018/12/23 13:18:29 Started spoofed metadata server
2018/12/23 13:18:29 Build id = localbuild_9cef1240-3a68-4ec3-a273-f49cd018316d
2018/12/23 13:18:29 status changed to "BUILD"
Starting Step #0 - "docker"
Step #0 - "docker": Already have image (with digest): gcr.io/cloud-builders/docker
Step #0 - "docker": Sending build context to Docker daemon   5.12kB
Step #0 - "docker": Step 1/3 : FROM busybox
Step #0 - "docker":  ---> 59788edf1f3e
Step #0 - "docker": Step 2/3 : ARG THE_SECRET
Step #0 - "docker":  ---> Using cache
Step #0 - "docker":  ---> f289a756b157
Step #0 - "docker": Step 3/3 : RUN echo "::${THE_SECRET}::"
Step #0 - "docker":  ---> Running in 0e90f8f4f349
Step #0 - "docker": ::This is the super secret secret.::
Step #0 - "docker": Removing intermediate container 0e90f8f4f349
Step #0 - "docker":  ---> 75d19dee1d47
Step #0 - "docker": Successfully built 75d19dee1d47
Finished Step #0 - "docker"
2018/12/23 13:18:35 status changed to "DONE"

It is worth noting that using build args for secrets is not recommended. Anyone with the image can see what the argument passed in was.

$ docker history 75d19dee1d47
IMAGE               CREATED             CREATED BY                                      SIZE                COMMENT
75d19dee1d47        4 days ago          |1 THE_SECRET=This is the super secret secre…   0B
f289a756b157        4 days ago          /bin/sh -c #(nop)  ARG THE_SECRET               0B
59788edf1f3e        2 months ago        /bin/sh -c #(nop)  CMD ["sh"]                   0B
           2 months ago        /bin/sh -c #(nop) ADD file:63eebd629a5f7558c…   1.15MB

Docker 18.09, added build secrets for a better solution, but GCB is still running Docker 17.12, so we will have to wait for that update.

A gist of the code is available at: https://gist.github.com/jmhobbs/a572b47048eb42803bcb2102ac57a8df