Zolmeister

JWT Authentication

2016-09-26T00:00:00-05:00

Authentication models for hosting platforms require more considerations due to 3rd party api access. We wanted to be able to issue user auth tokens for 3rd party apps (think OAuth), as well as be able to verify authentication of local native api calls (through our iframe native app hooks). Our solution was to use Json Web Tokens (JWT) with ES256 encryption (public/private key pair), which allows us to verify signed tokens using a public key which can be shared without compromising the security of our tokens. It also allows us to statelessly sign tokens for 3rd party developer use.

Here's a sample implementation of a basic authentication strategy (extensible to allow scoped tokens).

# auth_model.coffee
jwt = require 'jsonwebtoken'

module.exports =
  fromUserId: (userId) ->
    Promise.resolve {
      accessToken: jwt.sign {
          userId: userId
          scopes: ['*']
        }, config.JWT_ES256_PRIVATE_KEY, {
          algorithm: 'ES256'
          issuer: config.JWT_ISSUER
          subject: userId
        }
    }

  userIdFromAccessToken: (token) ->
    Promise.promisify(jwt.verify, jwt)(
      token,
      config.JWT_ES256_PUBLIC_KEY,
      {issuer: config.JWT_ISSUER}
    )
    .then ({userId} = {}) -> userId

And the corresponding Express middleware:

# auth_middleware.coffee
module.exports = (req, res, next) ->
  accessToken = req.query?.accessToken
  unless accessToken?
    return next()

  Auth.userIdFromAccessToken accessToken
  .then User.getById
  .then (user) ->
    if user?
      # Authentication successful
      req.user = user
    next()
  .catch next

MySQL Migrations

2016-08-27T00:00:00-05:00

Migrating 17M rows with no downtime - Isotope

Scenario: 17M rows in a table, constant writes/reads, slave/master setup.

If you try to run an ALTER TABLE (especially on older versions of MySQL), most operations will lock the table for writes for the entire run-time of the query. To avoid this, we need to incrementally update the table, incorporating write changes as they arrive.

Our solution was to use the Large Hadron Migrator, built and open-sourced by SoundCloud. LHM creates a copy table for the migration, with change triggers set up to keep up with new writes. It handles all the low-level details of the migration, and generally works well (though slowly, as I'll illustrate later).

First step is to create a migration.

$ apt-get -y install libmysqlclient-dev libmysqlclient18 ruby-dev build-essential rake
$ gem install activerecord-mysql-adapter standalone_migrations lhm
$ rake db:new_migration name=add_some_column_to_some_table
run/test locally: `RAILS_ENV=production rake db:migrate`
dump schema: `RAILS_ENV=production rake db:schema:dump`

Our database ran too slowly for the default change-rate of LHM (batches were too large, causing writes to time-out). By decreasing the speed of migration we avoided this, at the cost of migrations taking 20hr+ (we eventually moved away from MySQL).

# Throttle is in ms - default 100
# Stride is in number of rows - default 40,000
options = { :stride => 16, :throttle => 100 }

Next, we'll build a docker container to be able to run the migration remotely

FROM ubuntu:14.04

RUN apt-get -y update
RUN apt-get -y install libmysqlclient-dev libmysqlclient18 ruby-dev build-essential rake
RUN gem install activerecord-mysql-adapter standalone_migrations lhm

ADD . /opt/isotope
WORKDIR /opt/isotope

docker build -t isotope .
docker push isotope

Finally, to run the migration itself

docker run -e RAILS_ENV=production -e MYSQL_MASTER_HOST=<MYSQL PRODUCTION IP> -e MYSQL_MASTER_PASS=<PASSWORD> -i -t isotope rake db:migrate

If something goes wrong during the migration we have to clean up the change table manually.

drop `lhmn_*` tables
drop `lhmn_*` triggers

Clay.io - A Reflection

2016-05-04T00:00:00-05:00

Moving On

As of January I am no longer with Clay. I've learned so much over the past year and a half, more so than I could have ever imagined. Every minute working on Clay was a joy, and without a doubt one of the most fulfilling experiences of my life. As for my next adventure, stay tuned.

Lesson 1: Focus

Focus proved to be the most important lesson I've learned about running a company. It's so easy to get distracted and off track, without even realizing it. Focus means having concrete goals with timelines, and putting all major efforts towards those goals. Concrete goals are measurable and achievable, e.g. "Improve Day 1 retention by 1% in 1 month". Goals without a timeline (to measure ROI by time) or that aren't easily measurable (like happiness of users) should not be used.

At Clay we picked "Engaged Game Plays" (game plays lasting at least 1min) as our first KPI (key performance indicator). We made sure that all work being done would have a measurable impact on this metric. Had we been even stricter about only doing necessary work (e.g. we still did user support) I think we would have been even better off.

For most early stage companies I recommend retention (e.g. Day 1, Day 7) as the only key metric, because it's easy to measure, test, and is a relatively clear indicator of value. I also recommend that everyone at the company know the metric, and understand that their work is to align with that goal. Constant reminders (e.g. posters, screens, etc.) can go a long way to getting everyone on the same page.

Lesson 2: Validation

Failure should always be an option. Just because something should work, doesn't mean that it will. Any and all assumptions made about the problems you're solving need to be validated, and the process this takes is imperative to the growth of the company. This is where I think we could have improved the most.

Being explicit about the assumptions of the company vision, e.g. "Users would rather do X than Y", and validating as quickly as possible can save months of time and money (burn rate). This validation process can be difficult, complex, and/or expensive (to do properly), but how founders approach this problem can greatly improve chances of long-term company success.

Lesson 3: Science

Science was probably our biggest strength at Clay. We never really argued about anything; discussions were always focused around evidence-based metric growth. Even then nobody was ever "right" or "wrong" because we tried to A/B test all valid (high ROI) ideas. Science is all about proving (within statistical margins of error) assumptions correct or incorrect. We did this with nearly every feature we ever released, to both prove positive KPI impact and to calculate growth.

We ended up building our own A/B testing analytics framework (open-source) due to limitations (and cost) of other solutions. This was probably the hardest problem I solved at Clay, and even then I would like to take a second pass at it. I will likely write a blog post about it later this year.

Zorium v1.0.0 (╯°□°)╯︵ ┻━┻

2015-05-24T00:00:00-05:00

The CoffeeScript Web Framework

Zorium (source) is the web framework we use at Clay.io.
Today it has finally reached version v1.0.0 (╯°□°)╯︵ ┻━┻

This has been one of the most challenging projects I've worked on, with many great lessons learned. What follows is how Zorium came to be what it is today, and an explanation of design decisions in the framework.

With Zorium (or any framework), the code is the easy part. The hard part is the design and architecture of an application. This is where Zorium stands out from the rest. Just start with Zorium Seed.

For the full documentation: https://zorium.org

Special thanks to Matt Esch and Jake Verbaten, without whom Zorium would not exist.

Example

Because every framework should start with code:

z = require 'zorium'

class AppComponent
  constructor: ->
    @state = z.state
      name: 'Zorium'

  render: =>
    {name} = @state.getValue()

    z 'div.zorium',
      z 'p.text',
        "The Future is #{name}"

z.render document.createElement('div'), new AppComponent()
# <div class="zorium"><p class="text">The Future is Zorium</p></div>

Notice:

Components are just CoffeeScript classes
There is no magic pre-processor (i.e. JSX)
It's simple, intuitive, and idiomatic. No magic.

API Overview

The two most important methods in Zorium are z() and z.state().

z() is simply an extension of virtual-hyperscript, which understands Zorium components. The most difficult implementation detail here was constructing an in-memory cached tree for efficient rendering. Eventually this was solved with Thunks and traversing the virtual-tree (source).

# Component composition example
class Brick
  render: ({name}) ->
    z '.z-brick'.
      "Hello #{name}"

class House
  constructor: ->
    @$brick = new Brick()
  render: =>
    z '.z-house',
      z @$brick, {name: 'Zorium'}

z.state() creates an instance of Rx.BehaviorSubject(), with a set() method for diffing the current value (and also some lazy-ness to make async more efficient).

Originally a simpler model using observ-struct was used, however it did not provide a declarative syntax for dealing with complex streams of data.

e.g. We want to display the number of likes of the users favorite game.

If the user changes their favorite game, the value should update.
If another user likes the game, the value should update.

# Note that we assume the models update themselves and emit streams.
z.state {
  numLikes: Models.User.getMe().flatMapLatest ({favoriteGameId}) ->
      return Models.Game.getById favoriteGameId
    .map (game) ->
      return game.numLikes
}

Zorium Paper

https://zorium.org/paper

Zorium Paper was the first npm-installable set of components for Zorium. Here's what it looks like to use it:

# npm install zorium-paper
Button = require 'zorium-paper/button'

class Clicker
  constructor: ->
    @$button = new Button()
  render: ->
    z '.z-clicker',
      z @$button,
        $content: 'Click Me'
        isRaised: true

The key to using an npm module and maintaining efficient css (separable styles for production) is to use webpack. Webpack lets Zorium components declaratively define styles which can be efficiently loaded using webpack without resorting to inefficient inlining or other duplication.

Just having webpack is not enough though. Without shadow-dom we don't have true CSS isolation. In order to let developers use semantic classes without conflicting with each other, a namespace pattern was developed:

(full docs)

if window?
  require './index.styl'

class BigDrawer
  render: ->
    z '.z-big-drawer', # namespace
      z '.blue', 'blue'

// index.styl

// namespace
.z-big-drawer
  // direct children only
  > .blue
    background: blue

User-land components should use the z- prefix, while library authors should namespace under a different prefix.

Server Side Rendering

Efficient server-side rendering is non-trivial. It requires application code to be written statelessly, and thus needs to be considered from the start of a project. In fact, the state issue was so elusive, it wasn't until after Zorium v1.0.0-rc15 that I realized a critical mistake I had made regarding concurrent requests.

After realizing my mistake, I decided to take a page from React, and implement a simpler z.renderToString() (with async support).
Note that the server simply acts as a pre-renderer. Do not write server-side business logic here.

# server-side
app = express()
app.use (req, res, next) ->
  z.renderToString z new App(), {req, res}
  .then (html) ->
    res.send '<!DOCTYPE html>' + html
  .catch (err) ->
    if err.html
      log.error err
      res.send '<!DOCTYPE html>' + err.html
    else
      next err

The built in router, z.router, attempts to expose an express-like api. Note the difference in where new App() is called, as client-side you want a stateful App for transitions.

$app = new App()
z.router.use (req, res) ->
  {path, query} = req
  res.send z $app, {req, res}

Sometimes it will be necessary to split code between node and client-side. Use window? to check:

Promise = if window?
  window.Promise
else
  # Avoid webpack include
  _Promise = 'promiz'
  require _Promise

Zorium's boldest claim is probably that you can achieve nearly 100% code re-use for client and server-side code. Normally with other frameworks database requests are made and injected at different initialization points within the application.

This doesn't scale well for complex nested structures and requires duplicate effort in marshalling data. Zorium uses lazy-states and a platform-agnostic request library to create a seamless transition from client-side code to server-side code.

This area is probably the most complex because developers usually don't have to worry about state in this way. (e.g. creating singletons client-side is fine, but server-side will cause elusive caching bugs)

Comparison to other virtual-dom libraries

The Zorium framework is a collection of patterns, which are expressed in Zorium Seed. However, as a library is has some key differences from other popular virtual-dom libraries.

React - The Facebook-backed library that started it all.
- React source-code is a convoluted mess. Extending it is nearly impossible. [1]
- The Flux architecture is unnecessarily complex. [2]
- React still has some clear deficiencies with complex animations. [3]
- The React API makes questionable design choices. [4][5][6]
Mithril - A micro vritual-dom library which seemed quite promising.
- We initially used this at Clay for a few months (in a fork), but eventually got rid of it entirely.
- Source-code is quite complicated, making extending difficult.
- Manual state-management and rendering updates hurt scalability as our projects grew.
- Lack of structured best-practices lead to a lot of guessing and implementation mistakes.

Both React and Mithril use different virtual-dom implementations. Zorium uses the virtual-dom library, which is the standard behind many other virtual-dom frameworks (e.g. Mercury).

1: Seriously, this is incomprehensible. (vs z.render() for example)
2: Because Zorium supports RxJS Observables in state, async code is declaratively directional (and streaming) and thus avoids the data-binding issues Flux is said to address.
3: Inter-component communication with animations is nearly impossible (without an event-bus and side-effects), whereas Zorium simply uses RxJS observables to handle complex communications in a declarative way.
4: React.createClass() (and extending React.Component) means non-idiomatic initialization. (vs. explicitly calling new Component())
5: JSX magic is not idiomatic code.
6: Mixins are also a mistake, they are opaque and imperative.

10x: Service Discovery at Clay.io

2015-02-17T00:00:00-06:00

Architecture

Service Oriented Architectures can be one of the most iterable and available software configurations for building almost any product. There are many challenges that come with these systems, probably the biggest of which is service discovery. This is how your services will communicate with each other. For our service discovery, we turn again to Docker. If you haven't read how we do deploys: Docker at Clay.io

Tutorial

Synapse (https://github.com/claydotio/synapse) is a daemon which dynamically configures a local HAproxy configuration which routes requests to services within your cluster. It watches Amazon EC2 (or another endpoint) for services, specified in a configuration file. We use the ec2tag watcher, which easily lets us add servers to our cluster by tagging them.

HAProxy

Alright, so let's start with the services. They will be talking to each other via the local HAProxy instance. Because services run inside docker, we need to specify the host IP for services to look for services at a specific port.

Here, we use Ansible to pass the local IP and port to service running on that machine.

SERVICE_API=http://{ ansible_default_ipv4.address }:{ service_port }

For a service to use another service, they simply make calls to that IP/port. The key here is that the IP is the local machine IP, which is handled by HAProxy. We have released an HAProxy docker container which watches a mounted config file, and updates automatically on changes:
https://github.com/claydotio/haproxy

docker run
    --restart always
    --name haproxy
    -v /var/log:/var/log
    -v /etc/haproxy:/etc/haproxy
    -d
    -p 50001:50001
    -p 50002:50002
    -p 50003:50003
    -p 50004:50004
    ...
    -p 1937:1937
    -t clay/haproxy

By default, we use the noop config at /etc/haproxy, which gets mounted inside the docker container and watched for changes. We will be mounting the same haproxy config inside our synapse container in a moment. It's important to note that if this container goes down, all services on the machine will be cut off from all other services. For that reason, we have allocated additional ports to the container for use with new services in the future (as they cannot be dynamically allocated).

Synapse

Ok, now it's time to actually set up synapse.

Running synapse (thanks to our public Docker container) is easy.

docker run
    --restart always
    --name synapse
    -v /var/log:/var/log
    -v /etc/synapse:/etc/synapse
    -v /etc/haproxy:/etc/haproxy
    -e AWS_ACCESS_KEY_ID=XXX
    -e AWS_SECRET_ACCESS_KEY=XXX
    -e AWS_REGION=XXX
    -d
    -t clay/synapse
    synapse -c /etc/synapse/synapse.conf.json

Notice how we are mounting a synapse config, and an haproxy config inside the container. The HAProxy config is our noop config from before (because it will be auto-generated by synapse), but let's look into configuring synapse.

Configuring synapse can be a bit tricky, as the documentation could be better. Here is an example config that should explain everything that's missing in the docs:

source

{
  "services": {
    "myservice": {
      "discovery": {
        // use amazon ec2 tags
        "method": "ec2tag",
        "tag_name": "servicename",
        "tag_value": "true",
        // if this is too low, Amazon will rate-limit and block requests
        "check_interval": 120.0
      },
      "haproxy": {
        // This is the port other services will use to talk to this service
        // e.g. http://10.0.1.10:50003
        "port": 50003,
        "listen": [
          "mode http"
        ],
        // This is the port that the service exposes itself
        "server_port_override": "50001",
        // This is our custom (non-documented) config for our backup server
        // See http://zolmeister.com/2014/12/10x-docker-at-clay-io.html
        // for details on how our zero-downtime deploys work
        "server_backup_port": "50002",
        "server_options": "check"
      }
    }
  },
  // See the manual for details on parameters:
  // http://cbonte.github.io/haproxy-dconv/configuration-1.5.html
  "haproxy": {
    // This is never used because HAProxy runs in a separate container
    // Reloads happen automatically via the file-watcher
    "reload_command": "echo noop",
    "config_file_path": "/etc/haproxy/haproxy.cfg",
    "socket_file_path": "/var/haproxy/stats.sock",
    "do_writes": true,
    "do_reloads": true,
    "do_socket": false,
    // By default, this is localhost, however because HAProxy is running
    // inside of a container, we need to expose it to the host machine
    "bind_address": "0.0.0.0",
    "global": [
      "daemon",
      "user    haproxy",
      "group   haproxy",
      "chroot  /var/lib/haproxy",
      "maxconn 4096",
      "log     127.0.0.1 local0",
      "log     127.0.0.1 local1 notice"
    ],
    "defaults": [
      "log            global",
      "mode           http",
      "maxconn        2000",
      "retries        3",
      "timeout        connect 5s",
      "timeout        client  1m",
      "timeout        server  1m",
      "option         redispatch",
      "balance        roundrobin",
      "default-server inter 2s rise 3 fall 2",
      "option         dontlognull",
      "option         dontlog-normal"
    ],
    "extra_sections": {
      "listen stats :1937": [
        "stats enable",
        "stats uri /",
        "stats realm Haproxy Statistics"
      ]
    }
  }
}

Conclusion

That's all there is to it. Special thanks to Airbnb for open-sourcing their tool, which allowed us to set up service discovery in a simple and scalable way. For those not on Amazon EC2, there is a Zookeeper watcher (which we did not want to deal with), and hopefully soon an etcd watcher:
https://github.com/airbnb/synapse/pull/58

Once that's merged, we may move to using Nerve with etcd instead of EC2 tags to handle service announcement. For reference, I'll leave this etcd example docker information here:

curl https://discovery.etcd.io/new?size=3
docker run
    --restart always
    --name etcd
    -d
    -p 2379:2379
    -p 2380:2380
    -v /opt/etcd:/opt/etcd
    -v /var/log:/var/log
    -v /etc/ssl/certs:/etc/ssl/certs
    quay.io/coreos/etcd:v2.0.0
    -data-dir /opt/etcd
    -name etcd-unique-name
    -listen-client-urls http://0.0.0.0:2379
    -listen-peer-urls http://0.0.0.0:2380
    -advertise-client-urls http://localhost:2379
    -initial-advertise-peer-urls http://localhost:2380
    -discovery https://discovery.etcd.io/XXXXXXXXXXXX
    -initial-cluster-token cluster-token-here

(see discovery docs for url)

10x: Docker at Clay.io

2014-12-15T00:00:00-06:00

Intro

At Clay, our deploy process is fun. Seriously. Because we use Docker.

Docker is a containerization tool that allows for easy application isolation and deployment. The basic idea is that Docker runs your application in a virtualized isolated environment, similar to a virtual machine, but without the overhead. You start with a 'base image', and then describe how to create a 'container' using a Dockerfile.

Overview

Update: Example Ansible Config (may not follow best practices)

Our deploy process looks like this:

Merge all code to be shipped into git master branch
git tag for release (eg. v1.0.0)
Build docker image (code lock)
docker tag for release (eg. v1.0.0)
Push docker image to the registry
Update staging cluster container versions (zero downtime)
- Make sure backup container is running
- Upgrade master container
- Wait for new master to be up
- Upgrade backup container
Update production cluster container versions (zero downtime)
- (see above process)

This process guarantees that we are running the exact same code in staging and production, and allows us to roll back to the last docker container release version if anything goes wrong.

Clay.io Dockerfiles

Here is an example Dockerfile from our mobile app (source):

FROM dockerfile/nodejs:latest

# Install Git
RUN apt-get install -y git

# Add source
ADD ./node_modules /opt/clay-mobile/node_modules
ADD . /opt/clay-mobile

WORKDIR /opt/clay-mobile

# Install app deps
RUN npm install

CMD ["npm", "start"]

The file is self-explanatory. It simply copies the source code from the current directory into the container. Environment variables will be used to introduce sensitive / dynamic configuration. One important note is that npm start actually compiles and minifies the code for production:

// package.json
{
  "scripts": {
    "build": "node_modules/gulp/bin/gulp.js build"
    "start": "npm run build &&
              ./node_modules/pm2/bin/pm2 start ./bin/server.coffee
                -i max
                --name clay_mobile
                --no-daemon
                -o /var/log/clay/clay_mobile.log
                -e /var/log/clay/clay_mobile.error.log",
  }
}

This is because we need to re-build the static files with the production environment variables. For more examples, checkout our GitHub: github.com/clay.io

Basic Deployment of A Docker Image

At Clay, we host our images on the docker registry.

Because of Docker, deploying our applications to staging and production environments is trivial. This is the entire process (untagged container, to staging - automated with Ansible):

# Local machine / Build server
docker build -t clay/mobile .
docker push clay/mobile

# Staging / Production server
docker pull clay/mobile
docker run
    --restart on-failure
    -v /var/log/clay:/var/log/clay
    -p 50000:3000
    -e CLAY_MOBILE_HOST=XXXX
    -e CLAY_API_URL=XXXX
    -e FC_API_URL=XXXX
    -e NODE_ENV=production
    -e PORT=3000
    --name mobile
    -d
    -t clay/mobile:VERSION

(Note: Ports 49,152 - 65,535 are generally used for private applications)

Zero-Downtime Updates

If you noticed in the above start script, we use PM2 to handle clustering multiple server processes. PM2 supports zero-downtime code updates, however because it is inside the container and we never change code within a container at runtime, we don't use it. PM2 is used simply to take advantage of multiple server cores.

The key to zero-downtime updates is to run two server processes. A master process, and a backup process. We do this by assigning different ports to two container deployments:

docker run ... -e PORT=50000 --name mobile
docker run ... -e PORT=50001 --name mobile-backup

HAProxy handles re-routing traffic to the backup server when the master goes down:

# example haproxy.cfg

backend mobile
  mode http
  balance roundrobin
  server app1 x.x.x.x:50000 check
  server app1b x.x.x.x:50001 check backup
  server app2 x.x.x.x:50000 check
  server app2b x.x.x.x:50001 check backup

A future post on HAProxy will go into more details on how we use HAProxy to load balance across our servers.

The following deploy process is fully automated by Ansible:

Verify backup healthy container status (Ansible)
docker pull clay/mobile:v1.0.0
Kill master container (network requests re-route automatically to the backup)
- docker rm -f mobile
Update master container, and restart
- docker run ...
Once the master container comes back up, network requests will move back to the master
Kill and update the backup container
- docker rm -f mobile-backup && docker run ...

If anything goes wrong, simply revert to an older image version:
docker run -t clay/mobile:v0.0.12.

Closing

If you missed previous 10x posts:

10x: Logging at Clay.io

2014-10-25T00:00:00-05:00

Managing 20+ servers as a small team is no easy task, and when things go wrong (they always do) figuring out what happened quickly is essential. Of course we can't ssh into each machine, that would take ages, so instead we use Logstash to aggregate our logs.

This is the second post in my 10x series, and if you missed last episode: Architecture at Clay.io.

Logstash overview

Logstash deployments have two parts. The aggregate server (or cluster), and the client servers. The aggregate server runs the main logstash binary, while the client servers run logstash-forwarder to ships logs to the logstash instance. Here is a diagram of our setup:

Note that logstash-forwarder doesn't rotate logs. Log rotation is the process of storing older logs (usually compressed) into buckets (either of size or by date) and then eventually expiring them. The (aptly named) tool for the job here is the logrotate linux utility.

Investigating logs

The logstash daemon runs ElasticSearch and Kibana to analyze logs. You can apply many filters and create complex queries into your log data this way because of ElasticSearch.

Note that you cannot expose ElasticSearch ports publicly or your server will get hacked (!). This happened to us, and our solution was to move our infrastructure to Amazon VPC so that the only way to get to the logs is to be inside of our privileged network.

Journalist + Scribe

At Clay.io we have open-sourced our Docker containers so that anyone can deploy a distributed logging system themselves in just a few seconds.

Logstash Server: clay/journalist-public
Logstash-forwarder Server: clay/scribe-public

Good Luck!

10x: Architecture at Clay.io

2014-10-01T00:00:00-05:00

This is the first post in my new series 10x, where I share my experiences and how we do things at Clay.io to develop at scale with a small team.

Update:

The Cloud

CloudFlare

CloudFlare handles all of our DNS, and acts as a distributed caching proxy with some additional DDOS protection features. It also handles SSL.

Amazon EC2 + VPC + NAT server

Almost all of our servers live on Amazon EC2, most are either medium or large instances. We also use Amazon VPC to host some of our servers inside of a private network, inaccessible from the outside world. In order to get into this private network we have a NAT server, which also serves as our VPN endpoint which we use when working with our internal network. (guide, OpenVPN)

Amazon S3

We use Amazon S3 as our CDN backend, which hosts all of our static content. We use a separate domain for this: cdn.wtf for security and performance reasons (cookie-less domain).

HAProxy

HAProxy is an extremely performant reverse-proxy which we use to route traffic to our different services. This work is non-trivial due to the nature of Clay.io and its platform support concerns (and legacy code support), which I will go into detail in a later article.

We currently have a single HAProxy server on an m3.medium instance, but will upgrade as traffic increases. In addition, we may add Amazon ELB in front to scale horizontally if necessary.

App Server - Docker

Docker is tool to manage Linux containers, which are similar to Virtual Machines except with less overhead (and without some isolation and security guarantees). The key to Docker is that code shipped inside of a container should run the same no matter what the host machine looks like.

We currently run most of our computational services on an app server via Docker. This server can easily be replicated to meet elastic demand, and services can be moved on and off easily. Eventually we hope to manage these app servers with a tool like Kubernetes. (See bottom of post)

Staging App Server - Docker

Our staging environment server is identical to our application server, and runs the exact same docker binaries that we run in production. This environment has been critical to preventing unnecessary breakage and downtime of our production systems.

Data

MySQL

MySQL is a production-hardened relational SQL database. The vast majority of our data currently resides inside a Master-Slave MySQL cluster. We have one master, and two slave servers which serve most of our queries for our users. Eventually we may have to move tables or shard the single master server, but hopefully not for a while.

Logstash

Logstash is a log aggregation tool, with Kibana integration for analysis. It basically handles all of our application logs, and gives us a place to check for errors when something goes wrong. It saves us from having to SSH into a machine to check logs.

MongoDB

MongoDB is a NoSQL document storage database. We currently use mongodb for some of our developer endpoints, and for our A/B testing framework Flak Cannon.

Memcached

Memcached is a key-value store, used mostly for caching. In many ways it is similar to Redis. We currently use Memcached in our legacy webapp for caching MySQL query results. Eventually we would like to replace this with Redis.

DevOps

Ansible

Ansible has been our tool of choice for managing our servers. It's simple enough for most developers to learn quickly and be comfortable working with, and has been critical for automating many of the processes normally managed by an operations team.

Other Services

GitHub

GitHub - Great source code management, enough said.

Uptime Robot

Uptime Robot is a free monitoring service which we use to monitor our healthchecks and endpoints. If anything goes down, it will email and text message us within 5min.

Drone.io

Drone.io is a continuous integration service, which we use to continuously run our test suites for our various projects. It is similar to TravisCI, and has recently released an open source self-hostable version.

Docker Registry

We currently use the official Docker registry to manage our docker containers. It's similar to GitHub, except for Docker containers.

New Relic

New Relic is a server and application monitoring service, which we mostly use to monitor our servers to let us know when a machine is running out of disk or memory

Google Analytics

Google Analytics is our main website analytics tracking tool. For tracking our site specific features, we use the custom events features.

Google Apps

Google Apps provides email for our domain clay.io, and gives our organization a shared Google Drive setup.

Last Pass

Last Pass is a password management service which allows us to share company credentials for all of the other above services easily amongst the team.

The Future

While we are currently happy with our setup today, we hope to improve it in the coming months. This initial infrastructure version is missing some features which weren't necessary enough to justify spending time on, but we will eventually need to come back to them as we scale.

Kubernetes is looking to be an amazing project and tool for managing Docker containers at scale. We will be following it's development closely and hopefully put it into production as the project matures.

Amazon Glacier is another technology we have been looking at for doing database backups, and hope to implement that in the near future.

RethinkDB, while quite immature, is also quite an interesting project. We will definitely be following it's progress and may eventually move some of our data into it as we move away from MySQL.

Developing on Linux

2014-09-13T00:00:00-05:00

dotfiles

i3 - Tiling Window Manager

A tiling window manager is a type of window manager that arranges your windows as tiles, usually in such a way that makes managing them more efficient via keyboard shortcuts. i3 is my window manager of choice, after having used xmonad for a long time, i3 provides more flexibility and ease of configuration.

Here is a great sceencast which will help you understand the power of tiling window managers. i3 does a great job of managing multiple workspaces, even across multiple monitors. While developing I actively use 7 distinct workspaces across two monitors (soon probably three) and having an efficient way of transitioning (alt-#) is critical to maintaining my workflow speed.

Here is a bit of config that I added to i3 (see dotfiles for the full version)

# Zolmeister
# disabled status bar (see above)
# screen lock with ctrl + sup + l
bindsym Control+mod4+l exec i3lock -c 000000 -i ~/Pictures/Z/Z-bg.png
# set border to 1
new_window 1pixel
# hide border if 1 window
hide_edge_borders both
# Resize Containers, Vim-style                            ($mod+Control+[hjkl])
bindsym $mod+Control+j resize grow height 5 px or 5 ppt
bindsym $mod+Control+k resize shrink height 5 px or 5 ppt
bindsym $mod+Control+l resize grow width 5 px or 5 ppt
bindsym $mod+Control+h resize shrink width 5 px or 5 ppt

# custom colors
# class                 border  backgr. text    indicator
client.focused                #1793D0    #1793D0    #FFFFFF
client.focused_inactive    #333333    #333333    #999999
client.unfocused            #333333    #333333    #999999
client.urgent                #FF0000    #8C5665    #FF0000

# background image
exec --no-startup-id nitrogen --restore

# fix media keys and gtk theme issues
exec gnome-settings-daemon &

# fix deb file installation
exec /usr/lib/policykit-1-gnome/polkit-gnome-authentication-agent-1 &

ZSH + Powerline Shell

Most shells today run [Bash](http://en.wikipedia.org/wiki/Bash(Unixshell) however ZSH is an alternative which has great autocompletion and other nifty feature For my .zshrc file, check my dotfiles.

In addition, the ZSH framework oh-my-zsh provides some amazing features and shortcuts, definitely work checking out.

Powerline shell is a beautiful shell extension which takes care of drawing your prompt. I have customized it based on my ZSH theme to include additional git hints (like ±).

Atom.io

Atom.io is without a doubt the best editor I have ever used. I have used Aptana (good, but bloated), Sublime Text (excellent, but not open source), and Brackets (ok, missing quite a few features) extensively. None of them compare to Atom (I refuse to use an editor that is not open-source, as my editor is a core part of my development workflow and I must have full control over it).

Some day I may switch to VIM/Emacs, but have been quite happy with my working speed in Atom.

Hardware

After joining Clay.io I have finally gotten around to building another desktop. The only spec requirements I had were a Linux friendly GPU (Nvidia), 32GB of RAM, and 4k monitor support.

Here is the build (newegg wishlist, missing PSU + case):

Fuco - A Coloring Game

2014-09-01T00:00:00-05:00

Welcome, to Fuco (Source), a coloring game! Fuco was made in 48 hours as part of the Ludum Dare game compo. The theme was Connected Worlds, which game me the inspiration for Fuco. Originally I had an idea for a game that mixed game art styles (pixel art + isometric + full 3D), however that was far too ambitious considering the time constraints and my artistic skills.

This was my second Ludum Dare, my first game being Nent which took advantage of WebGL shaders for a metaballs effect. Since I joined Clay.io I figured that this time I would make a game that could be played on mobile, which meant no WebGL (due to iOS support). Because I wrote platform-agnostic code, I was easily able to publish the game to many platforms using Cordova

Being my second time participating I should have had better time management. However I made most of the game on Sunday (pulling an all-nighter, being awake during the earthquake at 3am):

Negative coloring without shaders

When you play the game, you'll notice that coloring the correct area below will erase the color above, and an incorrect coloration will override the above color (incorrectly). Solving this problem in a performant way on HTML5 Canvas was not immediately obvious. If I had been able to use WebGL, I could easily write a shader which could do calculations based on each pixel value. However canvas is too slow to update each pixel each frame which meant my solution had to be based on native canvas calls.

Each section of the circle is defined by a closed shape, so I simply erase the region the player is brushing using ctx.clip() which allows me to mask draw calls. Here is my solution:

if (y < 0) {
  // erase inside selected brush clip region
  ctx.save()
  ctx.beginPath()
  region.path(ctx)
  ctx.clip()

  radgrad = ctx.createRadialGradient(x,y,brushSize/4|0,x,y,brushSize/2|0)
  radgrad.addColorStop(0, $white.toString())
  radgrad.addColorStop(0.9, $white.alpha(1).toString())
  radgrad.addColorStop(1, $white.alpha(0).toString())
  ctx.fillStyle = radgrad
  ctx.fillRect(x-brushSize/2|0, y-brushSize/2|0, brushSize, brushSize)
  ctx.fillRect(x-brushSize/2|0, y-brushSize/2|0, brushSize, brushSize)

  ctx.restore()
}

Home screen animation

One of my favorite effects I made for this game was the home screen animation of the logo. Some googling for SVG animation landed me at this example, which I adapted slightly for my purposes. I used Inkscape to draw the SVG paths, and them inlined the SVG into the HTML.

var $paths = document.getElementsByTagName('path')

// This is hard coded because it crashes android if you calculate it
var pathLengths = [114.63752746582031, 135.55255126953125, 115.6463851928711, 147.36000061035156, 3442.748291015625, 1806.394775390625, 8450.58984375, 8450.58984375]
window.addEventListener('load', function () {
  var i = $paths.length
  while(i--) {
    // This line will crash android browser clients
    // pathLengths[i] = $paths[i].getTotalLength()
    simulatePathDrawing($paths[i], i)
  }

  document.body.style.visibility = 'visible'
})

function simulatePathDrawing(path, i) {
  var length = pathLengths[i]
  path.style.transition = path.style.WebkitTransition = 'none';
  path.style.strokeDasharray = length + ' ' + length
  path.style.strokeDashoffset = length
  path.getBoundingClientRect()
  path.style.transition = path.style.WebkitTransition = 'stroke-dashoffset 1.25s ease-in-out'
  path.style.strokeDashoffset = '0'
  path.style.strokeWidth = '2px'
}