I needed to make a secure upload system and my first choice was Ruby of course since it’s where my team is the strongest. My design for the secure file upload system is roughly the following:
- A service calls the upload service to get a link. The call is made with a few bits of info, s3 bucket and path, link expiry, metadata, and a challenge array. The challenge array is used to verify the user getting the upload link is who they say they are without being authenticated. This could be a password, email, last 4 of social, etc…
- The information is serialized into a string then encrypted.
- The encrypted string is packed into a url safe base64 encoded string.
- When the user uses the link, the token is consumed by the service and the information is decrypted. The expiry is checked then the service uses the metadata and the challenge to produce a json schema or form that it sends back to the user.
- The user will solve the challenge and the information along with the token are sent back. The token is decrypted once again and the challenge is safely checked.
- If the challenge is successfully answered, then the service uses the information about the upload stored in the decrypted token to construct an S3 upload policy.
- The policy is then signed using AWS v4 signature SDK. The policy is sent back to the user and the Frontend uses it to construct a form.
- When the user adds documents to the page, the upload will go straight to S3 using the signed request.
- The s3 bucket has events turned on and a virus scanning service goes to work making sure the documents are clean.
- From there the rest is very boring.
What you’ll need
At first glance of the above design, I would reach for AWS lambda, and a dynamo db table, but that wouldn’t be any fun for the rest of my team. So I put the following restraints on myself.
- No Database
- Ruby Only
With those constraints in place, let’s look at the ingredients list.
- Framework: You’ll need Rails, Sinatra or whatever the cool kids are using these days.
- Crypto: You will also need a way to encrypt things. rbnacl is what I chose. It used the sodium native library behind the scenes.
- Cloud: Aws SDK for me thank you.
- Coffee: Light roast, organic and always direct trade.
Since the framework and AWS are well documented let’s move to the Crypto part. After my exhaustive research I decided that the builtin Crypto library was ‘unhip’ and that rbnacl was the way to go. Especially when I saw this in their docs.
NaCl puts cryptography on Rails! Instead of making you choose which cryptographic primitives to use, NaCl provides convention over configuration in the form of expertly-assembled high-level cryptographic APIs that ensure not only the confidentiality of your data, but also detect tampering. These high-level, easy-to-use APIs are designed to be hard to attack by default in ways primitives exposed by libraries like OpenSSL are not. — rbnacl docs
I’m not a complete idiot when it comes to security and cryptography, but I’m not an expert either. So using a library that takes the collective brain power of lots of security people and lets me focus on what I want to do with my service, sounds great! So I cracked my knuckles and installed rbnacl and rbnacl-libsodium (installs sodium for you and tweaks rbnacl for you to look on the gem path for it by monkey patching stuff like all good Ruby libraries should).
First Tip: Gemfile order
Rails loads your gems from top to bottom. So make sure if you are wanting rbnacl-libsodium to install sodium for you that it is the first gem in your Gemfile to also have rbnacl as a dependency. This is really important if you use the jwt gem, since it requires rbnacl too.
Second Tip: How to Store your Private Key
If you are like me, you wanted to use rbnacl to generate a private key. And if you are like me you want to actually be able to encrypt/decrypt stuff with it. In order to do this, you will need to store the key somewhere. I chose to have each environment of my service have a specific private key. At first I had trouble storing it.
I started reading through the docs, not the rbnacl docs, but the Ruby docs on string encoding and eventually landed on this nice little solution.
Third Tip: Using Nonces without a db
So I had my key and I’m encrypting things right an left, but I ran into a snag. Using the secure box method you need to create a nonce before you encrypt stuff. This nonce is a one time use thing. It is meant to ensure perfect forward secrecy. In this case, that means any time you generate a link, even if it’s the exact same payload being encrypted, the resulting encrypted token will be different. This also means you have to keep track of the silly little nonce. Typically I would do this with a redis, mongo, dynamo db, cockroach db, rethink db, raven db, your mom db, why are there so many databases db, or heck even a mysql if we are gonna get crazy. Wait, I can’t use a database! This application needs to be stateless. Why? Because I feel like it okay. And it lends itself to super simple deployments and coordination. So here is what I did. Using my private key from tip 2, and a nonce. I wrote some encryption helpers that will do the following. I have removed a few things that don’t matter, but in essence, and with a better design pattern, the below is what is happening.
Well, that’s all folks. I hope its helpful