Jekyll2022-05-09T21:50:22+00:00https://davidudelson.com/David UdelsonDavid Udelson's Corner of the InternetDavid UdelsonGSoC Update: Week 62017-07-08T00:00:00+00:002017-07-08T00:00:00+00:00https://davidudelson.com/blog/2017/07/08/gsoc-week-6<p>This week in GSoC: REST APIs and their specifications!
<!--more-->
The former half of the week I was travelling for the 4th of July, but the latter
half of week I started writing the specification for the REST API. I learned
that there is basically no standard for specificying a REST API, and actually
there’s quite a bit of disagreement on how this should be done. I also learned
that the endpoints of the API are supposed to be nouns, so the endpoints I
proposed in my proposal are completely off the mark. I don’t have enough of the
specification complete to submit it for review yet, but I hope to accomplish
this next Monday.</p>David UdelsonThis week in GSoC: REST APIs and their specifications!GSoC Update: Week 52017-07-01T00:00:00+00:002017-07-01T00:00:00+00:00https://davidudelson.com/blog/2017/07/01/gsoc-week-5<p>Notes on some blockers I encountered over my family vacation.
<!--more-->
Week 5 went by in a blur, mostly because I was at Disney World with my
family (Disney is exhausting!). Due to this I was not able to give much time to
GSoC, but I did spend a little bit of time working on the <code class="language-plaintext highlighter-rouge">read</code> endpoint for
the API. From this work I have a couple of blockers I am hoping to work through
with my mentor next week:</p>
<ol>
<li><a href="https://github.com/mirage/irmin/blob/master/src/irmin-http/irmin_http_server.ml#L306">irmin_http_server.ml l.306-309</a>
contains calls to the functions <code class="language-plaintext highlighter-rouge">contents_t</code>, <code class="language-plaintext highlighter-rouge">node_t</code>, <code class="language-plaintext highlighter-rouge">commit_t</code>, and
<code class="language-plaintext highlighter-rouge">branch_t</code>. What is the point of these functions? It sort of seems like
you’re making one type of store out of another, but I don’t understand how
the stores are different.</li>
<li>I don’t understand how it is possible to call <code class="language-plaintext highlighter-rouge">rd.Wm.Rd.foobar</code>
(e.g.
<a href="https://github.com/mirage/irmin/blob/master/src/irmin-http/irmin_http_server.ml#L265">here</a>).
<code class="language-plaintext highlighter-rouge">Wm</code> was defined earlier in the module, so shouldn’t it be simply <code class="language-plaintext highlighter-rouge">Wm.Rd.foobar</code>?</li>
</ol>
<p>Hopefully after I get these blockers out of the way, endpoint implementation
should come a lot easier.</p>David UdelsonNotes on some blockers I encountered over my family vacation.GSoC Update: Weeks 3 and 42017-06-24T00:00:00+00:002017-06-24T00:00:00+00:00https://davidudelson.com/blog/2017/06/24/gsoc-weeks-3-and-4<p>The first month of GSoC is already over! Time for some updates on my progress.
<!--more-->
Over the past two weeks I’ve gotten myself up to speed with using
<a href="https://github.com/inhabitedtype/ocaml-webmachine">webmachine</a>,
which required learning about OCaml
<a href="https://realworldocaml.org/v1/en/html/classes.html">classes</a>
and
<a href="https://github.com/inhabitedtype/ocaml-webmachine">objects</a>.
After getting these OCaml concepts under my belt, I wrote a few webmachine
examples in order to understand the standard way of interacting with the
library. At this point, I’m ready to start implementing the endpoints from my
<a href="https://docs.google.com/document/d/1Vu97h2gNuJMTEYdIcghFfo_9L4GtCpEprzWfOivhy6g/edit">proposal</a>
one-by-one, which I plan to start this week.</p>David UdelsonThe first month of GSoC is already over! Time for some updates on my progress.GSoC Update: Weeks 1 and 22017-06-14T00:00:00+00:002017-06-14T00:00:00+00:00https://davidudelson.com/blog/2017/06/14/gsoc-weeks-1-and-2<p>It’s already week 3 of Google’s Summer of Code! Time for an update on what I
accomplished in the first two weeks and what I’m looking to get done with the
rest of this week.
<!--more-->
In essence the past two weeks have been all about getting comfortable with the
project. This includes (in order from general to specific) OCaml, unikernels, the Mirage OS
project, and Irmin. Arguably all of this should have been done during the
community bonding period, and I definitely feel that since returning from my
study abroad I’ve been racing to make up for lost time in this regard.
Here’s a list of the things I’ve been up to the past two weeks:</p>
<ul>
<li>
<p>I started reading about OCaml features that are still mysterious to me, but
that I will need to contribute to Irmin effectively. These include:</p>
<ul>
<li><strong>polymorphic variants</strong> These are used all over the place in large OCaml
codebases, and Irmin is no exception. I’ve been reading about them in RWO
and have opened threads on
<a href="https://github.com/mirage/irmin/issues/415#issuecomment-304460556">github</a>
and the
<a href="https://discuss.ocaml.org/t/polymorphic-variants-and-module-signatures/302">ocaml forum</a>
to try and get a better grip on how they work.</li>
<li><strong>OOP</strong> Understanding OOP in OCaml is crucial to
understanding
<a href="https://github.com/inhabitedtype/ocaml-webmachine">ocaml-webmachine</a>,
which I plan to use to implement the
REST API. I’ve been reading about these in RWO as well.</li>
<li><strong>GADTs</strong> While definitely less common than polymorphic variants, these
are also used in Irmin. The best
<a href="https://wiki.haskell.org/GADTs_for_dummies">GADT tutorials</a>
I could find were written
in Haskell, so I checked something off my personal programming TODO list
and spent a few days during the first
week <a href="http://learnyouahaskell.com/">learning Haskell</a>.
Due to the large number of things I am trying to get familiar with in
order to become an effective intern, I was not able to make as thorough a
job of understanding Haskell as I would have liked (monads and transformer
stacks are still mysterious), but now I understand enough to get the gist
of the tutorials. (Update: I have since found a
<a href="http://mads-hartmann.com/ocaml/2015/01/05/gadt-ocaml.html">great GADT tutorial in OCaml</a>.)</li>
</ul>
</li>
<li>I consumed a large amount of online content related to unikernels and Mirage OS. This helped
me get a sense of where unikernel technology is (pretty new but already
<a href="https://mirage.io/blog/bitcoin-pinata-results">really cool</a>),
how it’s received by industry
(<a href="https://blog.docker.com/2016/01/unikernel/">Docker seems interested</a> (and I
lerned that all of my mentors work for Docker)),
and how far we have to go before unikernels are a mainstream technology
(there’s a lot to do!).</li>
<li>I got some hands-on experience building and deploying unikernels by moving
this site from github pages to a unikernel on AWS. This took <em>way longer</em> than
I thought it would, but I got an appreciation for the state of unikernel
tooling (not great for people who aren’t developers), and at the end of it I
produced a <a href="https://davidudelson.com/blog/2017/06/13/aws-unikernel-guide/">guide</a>.</li>
<li>Last week I went to an OCaml hacking event at MIT! I was only there for an
hour but I got to meet some cool people. While travelling back from the
hacking event, I missed Mirage’s biweekly catchup (better luck next week).</li>
<li>I opened an <a href="https://github.com/mirage/irmin/issues/451">issue</a> about
improving Irmin’s documentation for beginners like myself. This is an effort I’m
hoping to sustain in parallel with my actual GSoC project over the course of
the summer.</li>
</ul>
<p>My plan for the coming week is to get a handle on the OCaml language features I
listed above so I can write the first iteration of the framework for handling
REST API requests.</p>David UdelsonIt’s already week 3 of Google’s Summer of Code! Time for an update on what I accomplished in the first two weeks and what I’m looking to get done with the rest of this week.Hosting Static Websites as Unikernels on Amazon Web Services2017-06-13T00:00:00+00:002017-06-13T00:00:00+00:00https://davidudelson.com/blog/2017/06/13/aws-unikernel-guide<p>Since being accepted as an intern with the <a href="http://mirage.io/">Mirage OS</a>
project, I’ve been doing a lot of tinkering
with <a href="http://unikernel.org/">unikernels</a>. In particular, I’ve become convinced
that hosting a static site as a unikernel has a lot of benefits over traditional
methods. A unikernel is
a <a href="https://somerandomidiot.com/blog/2014/08/11/attack-surface-area/">secure</a> way
to self-host your site, which means you
have <a href="https://somerandomidiot.com/blog/2014/08/14/my-content-is-mine/">control</a>
over your own content. But because unikernels are a relatively new technology
and are still largely experimental, building your own with zero prior knowledge
is hard. I’m hoping to lower that barrier to entry with this guide.
<!--more-->
Here’s what we’ll be doing:</p>
<ul>
<li>building a custom unikernel to serve our static site using the Mirage OS
libraries and build tool</li>
<li>fetching a TLS certificate for our domain using <a href="https://letsencrypt.org/">Let’s Encrypt</a></li>
<li>deploying our shiny new unikernel and certificate to <a href="https://aws.amazon.com/">Amazon Web Services</a></li>
</ul>
<p>In case I made it sound like a cakewalk just now, a word of caution: the
unikernel ecosystem is still pretty immature, and there’s a pretty high chance
that what worked for me is going to require some small tweaks on your part. If
you’re afraid of getting your hands dirty, you might want to wait until the
tooling matures a bit before attempting this. This guide will work best for you
if you:</p>
<ul>
<li>use some type of <em>64-bit</em> *nix. AFAIK this guide should be
easily followable for all linux and BSD users. I am less sure about OSX users.
Mirage OS does not have support for Windows.</li>
<li>want to host a static website. If you are looking to host some other,
non-static type of site, or are interested in unikernels for some other reason
entirely, you will probably still be able to make use of the later parts of
this guide (where we deploy the unikernel to AWS). However, the unikernel code
will look significantly different, and for that part you’re on your own.</li>
<li>already have a custom domain for your site. I will not cover how to purchase
and configure a domain name.</li>
</ul>
<p>Most software guides install all needed dependencies in the
first step, but since some of the tools we’ll be using are kind of fragile,
we’re going to install them one at a time, verifying that they work as expected
at each step. That way, if you need to do some tinkering to get things right at
any point, you can continue to follow the guide afterwards. But for the
brave/impatient, here’s what we’ll be using:</p>
<ul>
<li>Mirage OS, which requires OCaml</li>
<li><a href="https://certbot.eff.org/">certbot</a> (if you don’t have a TLS cert for your domain but want one)</li>
<li>An AWS account (free tier will suffice)</li>
<li>The AWS CLI</li>
<li><a href="https://github.com/GaloisInc/ec2-unikernel">ec2-unikernel</a>, which requires Haskell</li>
</ul>
<p>Let’s dive in!</p>
<h3 id="1-install-mirage-os">1. Install Mirage OS</h3>
<p>First, we’re going to build a unikernel that we can run and test locally. We
will be using the Mirage OS libraries and build tool to build our unikernels.
With Mirage, unikernels are written in OCaml, but as long as you’re okay using
my unikernel code, you don’t have to know any OCaml to follow this guide. If
you are completely new to unikernels, know that there are <a href="http://unikernel.org/projects/">other systems</a> that you
can use to do this. However, I have no idea if unikernels built using other
systems will run on AWS, so I suggest you stick to Mirage for now. Follow the
<a href="https://mirage.io/wiki/install">install instructions</a> to install OCaml, OPAM,
and Mirage, then run these commands to test your installation:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>$ git clone https://github.com/mirage/mirage-skeleton
$ cd mirage-skeleton/tutorial/hello
$ mirage configure -t unix
$ make depend # might take a while
$ make
$ ./hello
2017-06-02 19:47:56 -04:00: INF [application] hello
2017-06-02 19:47:57 -04:00: INF [application] hello
2017-06-02 19:47:58 -04:00: INF [application] hello
2017-06-02 19:47:59 -04:00: INF [application] hello
</code></pre></div></div>
<p>If you get output similar to above, then your installation works! If not, you
can always pop into the #mirage channel on IRC for help troubleshooting your installation.</p>
<h3 id="2-compile-a-unikernel-for-your-static-site">2. Compile a unikernel for your static site</h3>
<p>Now that we have Mirage installed, we can build a unikernel for our static site!
Our unikernel is composed of two OCaml source files, <code class="language-plaintext highlighter-rouge">config.ml</code> and
<code class="language-plaintext highlighter-rouge">dispatch.ml</code>, which can be found
<a href="https://github.com/dudelson/davidudelson.com/tree/4006d2cdc70aa712f0e8594a9606570fdbd2c5ff/_mirage">here</a>.
These files are nearly identical to
the ones in <code class="language-plaintext highlighter-rouge">mirage-skeleton/applications/static_website_tls/</code>, except I’ve
tweaked them a bit to handle URLs with a trailing slash.
Let’s say the static site you want to serve
is located at <code class="language-plaintext highlighter-rouge">~/mysite/content</code>. Save the OCaml files as <code class="language-plaintext highlighter-rouge">~/mysite/config.ml</code>
and <code class="language-plaintext highlighter-rouge">~/mysite/dispatch.ml</code>, respectively, then edit line 5 of <code class="language-plaintext highlighter-rouge">config.ml</code> to
point to <code class="language-plaintext highlighter-rouge">content</code>. You also need to copy the directory
<code class="language-plaintext highlighter-rouge">mirage-skeleton/applications/static_website_tls/tls</code> to <code class="language-plaintext highlighter-rouge">~/mysite/tls</code>. Because
our unikernel is configured to automatically redirect HTTP requests to HTTPS, it
needs some type of TLS certificate to build and run correctly. For now we’re
going to use the self-signed ones included with the mirage static site example,
but in the next step we’ll grab a real TLS certificate for our domain using
Let’s Encrypt. If you just want a HTTP-only server, you are free to modify the
code yourself, but you should really be using TLS! With the certificate copied,
now run:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>$ cd ~/mysite
$ mirage configure -t unix --net=socket --http=8080 --https=4433
$ make depend && make
$ ./https
2017-06-02 20:04:47 -04:00: INF [tcpip-stack-socket] Manager: connect
2017-06-02 20:04:47 -04:00: INF [tcpip-stack-socket] Manager: configuring
2017-06-02 20:04:47 -04:00: INF [https] listening on 4433/TCP
2017-06-02 20:04:47 -04:00: INF [http] listening on 8080/TCP
</code></pre></div></div>
<p>These commands should generate a lot of output, the last of which is displayed
above. You should now be able to open <code class="language-plaintext highlighter-rouge">http://localhost:8080</code> in your browser,
which will redirect you to <code class="language-plaintext highlighter-rouge">https://localhost:4433</code> and then block you from
viewing that because the TLS certificate we supplied isn’t trustworthy. Because
we’re trying to verify locally that our site works as expected, this time I
recommend overriding the block so you can view your site (in Firefox click
“Advanced > Add Exception”). Once you’ve verified that your site works as
expected, you’re (almost) ready to upload your site to AWS.</p>
<h3 id="3-fetch-a-tls-certificate-for-your-domain-using-lets-encrypt">3. Fetch a TLS certificate for your domain using Let’s Encrypt</h3>
<p>Because we’re using a self-signed TLS certificate right now, all browsers will
prevent their users from visiting our site as a security precaution. In order to
actually serve our website over HTTPS, we need a real, trustworthy certificate.
This used to mean paying a certificate authority money, but since last year the
Let’s Encrypt project has been giving them away for free. If you were already
self-hosting your site over HTTPS and thus already have a TLS certificate for
your domain, just copy it to <code class="language-plaintext highlighter-rouge">~/mysite/tls</code> and you can skip this step.
Otherwise, go ahead and follow the instructions for your distro on
the <a href="https://certbot.eff.org/">certbot</a> website to download certbot. Select
“None of the above” for the “Software” option. Once you have certbot installed,
there are a couple of ways you can go about actually getting your certificate.
If you are migrating to a unikernel from a self-hosted server, your simplest
option is probably to install certbot on the server and use <code class="language-plaintext highlighter-rouge">certbot certonly
--manual --preferred-challenges http</code>, which will establish your ownership over
your domain by having you enter commands on the server. If, however, you are migrating from
something like wordpress or github pages (my case) where you don’t have control
of the server, you’ll have to run <code class="language-plaintext highlighter-rouge">certbot certonly --manual
--preferred-challenges dns</code>, which establishes your ownership over your domain
by having you set certain DNS records. Note that both methods will record the IP
of the machine you requested the certificate from, which in the <code class="language-plaintext highlighter-rouge">dns</code> case might
be different from the IP you plan to host the unikernel from. If you do not want
the IP of your development machine recorded and associated with the certificate,
one thing you could do (probably, I haven’t tested it) is boot up a linux
instance on AWS, associate it with the amazon elastic IP you plan to use for
your unikernel, point your DNS records at the elastic IP, install a conventional
webserver and certbot on the instance, copy over your static site files and
serve the site, and then use <code class="language-plaintext highlighter-rouge">certbot certonly --manual --preferred-challenges
http</code> to get your cert. Your choice! In any case, make sure you set
<code class="language-plaintext highlighter-rouge">--config-dir</code>, <code class="language-plaintext highlighter-rouge">--work-dir</code>, and <code class="language-plaintext highlighter-rouge">--logs-dir</code> to writeable paths so that you
don’t have to run the script as root (these are given relative to the present
working directory, <em>don’t</em> use absolute paths), and <code class="language-plaintext highlighter-rouge">-d example.com</code> to specify the
domains you want certificates for. See the
<a href="https://certbot.eff.org/docs/using.html#getting-certificates-and-choosing-plugins">certbot manual</a>
if you are unsure or would like more information. So, to summarize, in my case I
ran:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>$ cd ~ && certbot certonly -d davidudelson.com -d www.davidudelson.com --manual --preferred-challenges dns \
--config-dir .local/letsencrypt --work-dir .local/letsencrypt/work --logs-dir .local/letsencrypt/logs
</code></pre></div></div>
<p>to generate my certificate. Once you have yours, it’s time to update our
unikernel’s tls certs to match. It’s a good idea not to move any files in the
directory that certbot placed your certs in, because this might prevent certbot
from updating your certs in the future. Instead, let’s make symlinks to the
<code class="language-plaintext highlighter-rouge">live</code> directory (which always contains the up-to-date certificates)
files:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>$ ln -s ~/.local/letsencrypt/live/example.com/privkey.pem ~/mysite/tls/server.key
$ ln -s ~/.local/letsencrypt/live/example.com/cert.pem ~/mysite/tls/server.pem
$ ln -s ~/.local/letsencrypt/live/example.com/chain.pem ~/mysite/tls/ca-roots.crt
</code></pre></div></div>
<p>Now you have a real TLS certificate to use with your unikernel!</p>
<h3 id="4-install-ec2-unikernel">4. Install ec2-unikernel</h3>
<p>Next we are going to install
the <a href="https://github.com/GaloisInc/ec2-unikernel">ec2-unikernel</a> tool,
which automates the process
of uploading and installing unikernels to EC2 as Amazon Machine Images (AMIs).
You need to have <a href="https://docs.haskellstack.org/en/stable/install_and_upgrade/">Stack</a>
and the <code class="language-plaintext highlighter-rouge">guestfish</code> command-line utility installed in order to build and run the
tool, so install those first if you don’t have them already. Next, run:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>$ cd ~ && git clone https://github.com/GaloisInc/ec2-unikernel
$ cd ec2-unikernel
$ stack init
</code></pre></div></div>
<p>Now normally we would run <code class="language-plaintext highlighter-rouge">stack build --install-ghc</code> to build the tool, but unfortunately at
the time of writing, this package is a little bit broken, and it doesn’t compile
with the default settings. If you’re reading this at least a few weeks after the
time of writing, try a <code class="language-plaintext highlighter-rouge">stack build --install-ghc</code> and see if it works, but if not, I’ll
demonstrate what I had to do to get it working. In essence I needed to use an
earlier version of the <code class="language-plaintext highlighter-rouge">amazonka-*</code> set of packages, which required me to
both choose a resolver for stack that had the
versions of the packages I wanted and rollback the tool to an earlier version:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>$ git checkout 4d28b4c
$ stack config set resolver "lts-7.22"
$ stack build --install-ghc
</code></pre></div></div>
<p>Once the tool builds correctly, the command <code class="language-plaintext highlighter-rouge">stack exec ec2-unikernel</code> should
print out some error message. It fails because we haven’t set up AWS yet, which
is what we’re gonna do next!</p>
<h3 id="5-setup-aws">5. Setup AWS</h3>
<p>There’s a couple things we need to do on AWS to allow ec2-unikernel to upload
our unikernels correctly:</p>
<ul>
<li>Generate an AWS credential keypair</li>
<li>Install and configure the AWS CLI</li>
<li>Make an S3 bucket for our unikernels</li>
<li>Create the “vmimport” role</li>
<li>Create a security group for our unikernels</li>
</ul>
<p>We’ll need to generate the keypair using the AWS web interface, but once we’ve
done that, the rest can be done from the command line thanks to the AWS
CLI. Of course, all of this assumes that you have an AWS account, so create one
first if you don’t already have one. AWS has a free tier that gives access to S3
and the least powerful instances on EC2 for one year. That is, you get a free
year of unikernels!</p>
<h4 id="generate-an-aws-credential-keypair">Generate an AWS credential keypair</h4>
<p>After your account is set up, log in to the control panel
and click on your name in the top-righthand corner followed by “My Security
Credentials”. Amazon will prompt you to create an IAM user; I didn’t bother.
Click on “Access Keys” > “Create New Access Key” and you’ll be shown a dialogue
box with your access and private keys. <strong>Do not close this dialogue box yet</strong>.
There’s no way to see your secret key again once you do. We’ll be giving this
keypair to the CLI in just a minute, but if you don’t want to wait you can copy
it into a text file or something just to be safe.</p>
<h4 id="install-and-configure-the-aws-cli">Install and configure the AWS CLI</h4>
<p><a href="https://docs.aws.amazon.com/cli/latest/userguide/installing.html">Install</a> the
CLI, then run <code class="language-plaintext highlighter-rouge">aws configure</code>. Enter your AWS access and private keys from the
previous step, and enter “us-west-2” for your region. AWS separates its servers into
<a href="http://aws.amazon.com/AWSEC2/latest/UserGuide/using-regions-availability-zones.html#concepts-available-regions">regions</a>
and all the configuration we’re doing is only valid for one region.
Unfortunately, due to a bug in ec2-unikernel, the only region that works right
now is Oregon, so that’s the one we have to use.</p>
<h4 id="make-an-s3-bucket-for-our-unikernels">Make an S3 bucket for our unikernels</h4>
<p>AWS S3 is just storage. S3 storage is allocated by “bucket”. The ec2-unikernel
tool we installed before will upload our unikernel to an S3 bucket and then
import it from the bucket into EC2. To make an S3 bucket, run:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>$ aws s3 mb s3://<bucket-name>
</code></pre></div></div>
<p>When choosing a bucket name, keep in mind that the bucket namespace is shared by
your entire region. Make sure to pick a memorable name that’s unique to you. For
example, I named my bucket “dudelson-unikernels”.</p>
<h4 id="create-the-vmimport-role">Create the “vmimport” role</h4>
<p>This is something that ec2-unikernel needs to be able to import our unikernels
from S3 into EC2. Follow the instructions
<a href="https://docs.aws.amazon.com/vm-import/latest/userguide/vmimport-image-import.html#vmimport-service-role">here</a>
to set it up. The json files
described in the article are located in the <code class="language-plaintext highlighter-rouge">ec2-unikernel/policies</code> directory.
Note that you have to edit the parts of <code class="language-plaintext highlighter-rouge">role-policy.json</code> that are red in the
instructions to be the bucket name you chose in the last step.</p>
<h4 id="create-a-security-group-for-our-unikernels">Create a security group for our unikernels</h4>
<p>EC2 instances are controlled by “security groups” which describe what ports
are open for inbound and outbound network traffic on the instance. The default
security group that is created when you launch an instance opens only port 22
so that the instance can be accessed via SSH. In our case, we cannot access our
static site unikernel via SSH and only want it to serve web traffic, so we’re
going to create a security group that opens only ports 80 and 443. Use the
following command:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>$ aws ec2 create-security-group --group-name "<name>" --description "<description>"
$ aws ec2 authorize-security-group-ingress --group-name "<name>" --protocol tcp --port 80 --cidr 0.0.0.0/0
$ aws ec2 authorize-security-group-ingress --group-name "<name>" --protocol tcp --port 443 --cidr 0.0.0.0/0
</code></pre></div></div>
<p>The group <code class="language-plaintext highlighter-rouge"><name></code> and <code class="language-plaintext highlighter-rouge"><description></code> can both be something as simple as
“unikernels”, since you’re not sharing the security group namespace with your
entire region.</p>
<h3 id="6-build-and-deploy-your-unikernel">6. Build and deploy your unikernel</h3>
<p>And now, the main attraction! We’ve put all the pieces in place, and now we can
finally deploy our unikernel to AWS. But before we do so, we need to rebuild the
unikernel to target the Xen hypervisor, which is what AWS uses. This looks very similar to what we did in
step 1:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>$ cd ~/mysite
$ mirage configure -t xen --dhcp=true
$ make depend && make
</code></pre></div></div>
<p>When we build for the xen hypervisor, we need to enable DHCP so that our
unikernel can find the default gateway provided by AWS when it’s launched on
EC2. Now we can upload it:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>$ cd ~/ec2-unikernel
$ stack exec ec2-unikernel -- -o `cat ~/.aws/credentials | grep id | cut -d " " -f 3` -w `cat ~/.aws/credentials \
| grep secret | cut -d " " -f 3` -b <bucket-name> ~/mysite/https.xen
</code></pre></div></div>
<p>We provide ec2-unikernel with our AWS keypair and the name of the S3 bucket we
created, which is all it needs to upload and import our unikernel as an AMI. If
the commands to get the AWS keypair don’t work on your system, you can just
enter the credentials manually.
This takes a while, so now would be a good time to get a drink or something.
When that’s done, the last line of output will be the ID of the new AMI. We use
that to fire up the instance from the command line:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>$ AWS_SEC_ID=`aws ec2 describe-security-groups | grep -C 1 <group_name> | grep GroupId | tr -d '" ,' | cut -d ':' -f 2`
$ aws ec2 run-instances --image-id <image-id> --count 1 --instance-type t1.micro --security-group-ids $AWS_SEC_ID
</code></pre></div></div>
<p>The <code class="language-plaintext highlighter-rouge"><image-id></code> should look like “ami-ABCD1234”. <code class="language-plaintext highlighter-rouge"><group_name></code> is the name of
the security group you created. This should spit out a bunch
of json about the instance. Wait about 60 seconds, then run <code class="language-plaintext highlighter-rouge">aws ec2 describe-instances | grep "running"</code>.
If your unikernel is up,
you should get a line of output back. If not, wait a little longer and try
again. Once you’ve confirmed that your instance is up and running, use <code class="language-plaintext highlighter-rouge">aws ec2
describe-instances | grep "PublicDnsName"</code> to find the URL that you can use to
connect to your website. Pop that in your browser and you should get a security
warning like we did in step 2. This is again because the TLS cert we generated
is for our domain name, not blahblah.compute.amazonaws.com. Put this nonetheless
means that you were able to successfully connect to your unikernel running on
AWS!</p>
<h3 id="7-maintenance">7. Maintenance</h3>
<p>As I eluded to at the beginning of the guide, one of the really nice properties
of unikernels is every time you make a change to your site, you rebuild the
entire software stack and upload a fresh image. This means that even if an
attacker compromises your system through your website, the damage will be undone
with the next image. Here are the steps you’ll perform to update your site by
deploying a new unikernel:</p>
<ul>
<li>rebuild your static site. I’m using <a href="https://jekyllrb.com/">jekyll</a> for this, so for me that’s <code class="language-plaintext highlighter-rouge">bundle
exec jekyll build</code>.</li>
<li>rebuild your unikernel</li>
</ul>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>$ cd ~/mysite
$ mirage configure -t unix --net=socket --http=8080 --https=4433 # test build; OR
$ mirage configure -t xen --dhcp=true # production build
$ make depend && make
</code></pre></div></div>
<ul>
<li>upload your unikernel to EC2</li>
</ul>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>$ cd ~/ec2-unikernel
$ stack exec ec2-unikernel -- -o `cat ~/.aws/credentials | grep id | cut -d " " -f 3` -w `cat ~/.aws/credentials \
| grep secret | cut -d " " -f 3` -b <bucket-name> ~/mysite/https.xen
</code></pre></div></div>
<ul>
<li>launch the unikernel, reallocate the EIP, and terminate the old unikernel</li>
</ul>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>$ aws ec2 describe-instances | grep "InstanceId" # get the old image ID
$ aws ec2 run-instances --image-id <image-id> --count 1 --instance-type \
t1.micro --security-group-ids "<security-group-id>" | grep "InstanceId" # get the new image ID
$ aws ec2 associate-address --instance-id <new-instance> --allocation-id <allocation>
$ aws ec2 terminate-instances --instance-ids <old-instance>
</code></pre></div></div>
<p>Of course I’ve <a href="https://github.com/dudelson/website/blob/69151e91ab1260404a66bd53479e7b3cff9e1828/deploy-unikernel">scripted</a> this entire process from stem to stern.</p>
<p>And that’s it! Go enjoy your new-and-improved unikernel life, tell your friends,
and maybe write me a build tool that will make this easier.</p>
<h3 id="8-assign-an-elastic-ip-and-configure-dns">8. Assign an elastic IP and configure DNS</h3>
<p>AWS elastic IPs allow you to allocate a public IPv4 address that you can assign
to a running EC2 instance. As in the previous step when we didn’t have an EIP
yet, the instance is given a public IP anyway, but it’s chosen from the pool of
available IPs for your region, and once the instance is terminated, its IP is
released back into the pool, meaning that the next instance you start will have
a different IP. But an EIP, once allocated, belongs to you, even
if you terminate one instance and start another. This means you can point the
DNS records for your domain at the EIP, and whenever you build and deploy a new
unikernel, you can reassign the EIP to point to it, keeping your domain up in a
hassle-free way. One warning though: If you allocate an EIP and leave it idle
(e.g. not assigned to a running instance), you’re charged hourly for it. So make
sure to deploy your new unikernel and reassign the EIP before terminating the
old one. With all that in mind, let’s allocate an EIP and assign it to our
instance:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>$ ALLOC_ID=`aws ec2 allocate-address --domain vpc | grep "AllocationId" | tr -d '" ,' | cut -d ':' -f 2`
$ INST_ID=`aws ec2 describe-instances | grep "InstanceId" | tr -d '" ,' | cut -d ':' -f 2`
$ aws ec2 associate-address --instance-id $INST_ID --allocation-id $ALLOC_ID
</code></pre></div></div>
<p>After assigning an EIP, update the type A DNS records for your domain to point to the EIP, and your site is live!</p>David UdelsonSince being accepted as an intern with the Mirage OS project, I’ve been doing a lot of tinkering with unikernels. In particular, I’ve become convinced that hosting a static site as a unikernel has a lot of benefits over traditional methods. A unikernel is a secure way to self-host your site, which means you have control over your own content. But because unikernels are a relatively new technology and are still largely experimental, building your own with zero prior knowledge is hard. I’m hoping to lower that barrier to entry with this guide.UVa 12086: Potentiometers2015-08-03T00:00:00+00:002015-08-03T00:00:00+00:00https://davidudelson.com/blog/2015/08/03/uva-12086-potentiometers<p><a href="https://uva.onlinejudge.org/external/120/p12086.pdf">UVa 12086: Potentiometers</a> is a classic <a href="https://en.wikipedia.org/wiki/Fenwick_tree">Fenwick Tree</a> problem.
<!--more-->
This was my first time solving a problem using a fenwick tree, so I have two things for people who are new to this data structure to keep in mind. First, be mindful of the fact that the fenwick tree requires 1-based indexing. If you pass any of the fenwick tree functions an index of 0, your program will end up inside an infinite loop. Second, in this particular problem, you are given a value to <em>set</em> the potentiometer to. This is different from what the fenwick tree function <code class="language-plaintext highlighter-rouge">adjust</code> does, because <code class="language-plaintext highlighter-rouge">adjust</code> takes an index and an <em>increment</em>. Thus, an vector or array is needed to store the values of the potentiometers so that an increment can be passed to the <code class="language-plaintext highlighter-rouge">adjust</code> function.</p>
<p>My solution:
<script src="https://gist.github.com/dudelson/b3ec31f570c13cbcd85c0d99cafc1420.js"></script></p>
<noscript><i>
This source code is included as a gist. Please enable javascript to view the
gist inline, or view it on github <a href="https://gist.github.com/dudelson/b3ec31f570c13cbcd85c0d99cafc1420">here</a>.
</i></noscript>David UdelsonUVa 12086: Potentiometers is a classic Fenwick Tree problem.UVa 12532: Interval Product2015-07-29T00:00:00+00:002015-07-29T00:00:00+00:00https://davidudelson.com/blog/2015/07/29/uva-12532-interval-product<p><a href="https://uva.onlinejudge.org/external/125/p12532.pdf">UVa 12532: Interval Product</a> involves a clever application of segment trees. Abridged problem statement: You are given a list of integers and a number of queries. Each query can either change one integer in the list, or ask for the sign of the product between a range of indicies. Print out a string with the answers to all the queries concatenated together.
<!--more-->
This problem can be solved with a <a href="http://www.geeksforgeeks.org/segment-tree-set-1-sum-of-given-range/">segment tree</a>, which is quickly becoming one of my favorite data structures. Since each query wants only the sign of the product, taking the actual product is not necessary. Instead, each index in the segment tree will store the sign of the product of the range it represents as a single <code class="language-plaintext highlighter-rouge">char</code>. This is implemented as two extra functions in the <code class="language-plaintext highlighter-rouge">SegmentTree</code> class, <code class="language-plaintext highlighter-rouge">combine</code> and <code class="language-plaintext highlighter-rouge">getSign</code>. <code class="language-plaintext highlighter-rouge">combine</code> combines the signs of the products of the two child nodes into the sign of the product of the parent node, and is used in the <code class="language-plaintext highlighter-rouge">build</code>, <code class="language-plaintext highlighter-rouge">query</code> and <code class="language-plaintext highlighter-rouge">update</code> methods. <code class="language-plaintext highlighter-rouge">getSign</code> is used when a single value in the segment tree is changed; it converts an integer in a char representing its sign.</p>
<p>When I wrote this solution this first time, I unknowningly did it with a buggy segment tree implementation, so my program often segfaulted in the beginning. However, once I debugged a few lines in the segment tree class, my only other issue was forgetting about the 1-based indexing used by the problem, a detail I seem to be forgetting a lot.</p>
<script src="https://gist.github.com/dudelson/11f9160cee9d1f3ff2d75584f64eee09.js"></script>
<noscript><i>
This source code is included as a gist. Please enable javascript to view the
gist inline, or view it on github <a href="https://gist.github.com/dudelson/11f9160cee9d1f3ff2d75584f64eee09">here</a>.
</i></noscript>David UdelsonUVa 12532: Interval Product involves a clever application of segment trees. Abridged problem statement: You are given a list of integers and a number of queries. Each query can either change one integer in the list, or ask for the sign of the product between a range of indicies. Print out a string with the answers to all the queries concatenated together.UVa 793: Network Connections2015-07-25T00:00:00+00:002015-07-25T00:00:00+00:00https://davidudelson.com/blog/2015/07/25/uva-793-network-connections<p><a href="https://uva.onlinejudge.org/external/7/p793.pdf">UVa 793: Network Connections</a> is a good UFDS implementation practice problem.
<!--more-->
This problem can be solved efficiently by using a UFDS (if you are not familiar with the UFDS, I wrote more about it <a href="http://udelson.me/blog/2015/07/24/uva-599-the-forrest-for-the-trees/">here</a>).</p>
<p>I did make one big mistake while solving this problem which involved how I read the input. I read the integer ids of the computers to be connected like this: <code class="language-plaintext highlighter-rouge">int a = s[2]-'0'</code>, where <code class="language-plaintext highlighter-rouge">s</code> is the entire line of input I had just read in. But this only works for one-digit numbers! Even worse, there’s no explicit error thrown when the numbers are greater than 1 digit; it just means that I more than likely end up reading some character that isn’t a number, which results in bizarre and intractible errors. I have to admit that I dislike using stringstream…</p>
<script src="https://gist.github.com/dudelson/a53d77bdcd8e09c3302b892893b61de3.js"></script>
<noscript><i>
This source code is included as a gist. Please enable javascript to view the
gist inline, or view it on github <a href="https://gist.github.com/dudelson/a53d77bdcd8e09c3302b892893b61de3">here</a>.
</i></noscript>David UdelsonUVa 793: Network Connections is a good UFDS implementation practice problem.UVa 11503: Virtual Friends2015-07-25T00:00:00+00:002015-07-25T00:00:00+00:00https://davidudelson.com/blog/2015/07/25/uva-11503-virtual-friends<p><a href="https://uva.onlinejudge.org/external/115/p11503.pdf">UVa 11503: Virtual Friends</a> simplified problem statement: Several people are friending each other on social media. Every time two people become friends, print out the current size of their social network.
<!--more-->
This is a classic ufds problem. That is, the friend groups of the people involved can be modeled as disjoint sets. Every time a pair of people friend each other, we join their sets and print its size. It is necessary to replace the arrays that would be used in an integer UFDS with maps in order to accomodate the strings.</p>
<script src="https://gist.github.com/dudelson/91163964b17919626df926bdbd04ed4e.js"></script>
<noscript><i>
This source code is included as a gist. Please enable javascript to view the
gist inline, or view it on github <a href="https://gist.github.com/dudelson/91163964b17919626df926bdbd04ed4e">here</a>.
</i></noscript>David UdelsonUVa 11503: Virtual Friends simplified problem statement: Several people are friending each other on social media. Every time two people become friends, print out the current size of their social network.UVa 10507: Waking up brain2015-07-25T00:00:00+00:002015-07-25T00:00:00+00:00https://davidudelson.com/blog/2015/07/25/uva-10507-waking-up-brain<p><a href="https://uva.onlinejudge.org/external/105/p10507.pdf">UVa 10507: Waking up brain</a> simplified problem statement: There are several areas of the brain, each of which is connected to other areas of the brain. All the regions but 3 are “asleep”, but if a region is connected to 3 awake regions for a year, that region wakes up. Given the connections between various parts of the brain and the 3 regions that are initially awake, determine how long the entire brain takes to wake up.
<!--more-->
I solved this problem using an adjacency list, since in this problem, the brain can be represented as a graph. I also kept a bitset, which recorded whether each section of the brain was awake or asleep. To figure out how long the brain takes to wake up, I processed the adjacency list in iterations, looking to see how many sleeping areas of the brain were connected to 3 awake areas each iteration. Those areas would “wake up” this timestep, so I put them in a vector (one mistake I made initially was updating the bitset immediately, which is not correct because the regions of the brain must be processed in discrete timesteps, that is, the state from the beginning of each timestep must be preserved until the entire data structure has been processed, then it gets updated all at once). If this vector was empty after processing the adjacency list, it meant that no new areas of the brain had woken up this year, and therefore the brain would never wake up (I also make the mistake of returning from the main method in this case, when instead I should have skipped to the next test case). Otherwise, the bitset of awake regions of the brain would be updated. This loop is terminated when the number of awake regions in the brain equals the number of total regions.</p>
<p>Once again I have noticed that even a small number of mistakes/typos (in this case I counted two, as described above) can take a long time to debug. Practice makes perfect.</p>
<script src="https://gist.github.com/dudelson/47fe5c022f6c3a698e8fd2d863764067.js"></script>
<noscript><i>
This source code is included as a gist. Please enable javascript to view the
gist inline, or view it on github <a href="https://gist.github.com/dudelson/47fe5c022f6c3a698e8fd2d863764067">here</a>.
</i></noscript>David UdelsonUVa 10507: Waking up brain simplified problem statement: There are several areas of the brain, each of which is connected to other areas of the brain. All the regions but 3 are “asleep”, but if a region is connected to 3 awake regions for a year, that region wakes up. Given the connections between various parts of the brain and the 3 regions that are initially awake, determine how long the entire brain takes to wake up.