Quite common requirements, right?

And given that Phoenix has pretty good code generators and LiveView now has an amazing live_file_input component that automates uploading files in an interactive way, it should be pretty easy to implement.

Well...

Keep reading to discover what I learned in the last couple of days trying to code this simple task.

The initial project and schema

Let's start by creating a new project with the latest Phoenix version. I use asdf for this:

# Add plugins
asdf plugin-add erlang
asdf plugin-add elixir
# Install them
asdf install erlang 25.3
asdf global erlang 25.3
asdf install elixir 1.14.4-otp-25
asdf global elixir 1.14.4-otp-25
# Install Phoenix 1.7
mix local.rebar --force
mix local.hex --force
mix archive.install hex phx_new 1.7.2 --force

We can now generate a project. I'll name mine mercury:

mix phx.new mercury --install

We'll use a simple migration and schema for a Product with two attributes: name and a list of images, stored as an array of strings.

Phoenix code generators

Our first attempt is to use the Phoenix code generators:

mix phx.gen.live Products Product products name:string images:array:string

We'll see that the generated code for the images attribute is an input field with type of select (lib/mercury_web/live/product_live/form_component.ex):

<.input
  field={@form[:images]}
  type="select"
  multiple
  label="Images"
  options={[{"Option 1", "option1"}, {"Option 2", "option2"}]}
/>

That's reasonable because the code generators have no way to know that we intended the array of strings to be image URLs and that the input should have a type of file and be supported by the live_file_input component.

As powerful as the generators are for CRUD pages, in this case, we need to do it by ourselves.

(Find the code for this part on the phoenix-code-generator branch of the git repo)

Official live_file_input docs

Let's write the code for file uploads manually following the official docs for live_file_input. We'll start from the generated code and we'll adapt it by adding the code from the official docs.

Add the following routes to router.ex:

  scope "/", MercuryWeb do
    pipe_through :browser

    get "/", PageController, :home

    live "/products", ProductLive.Index
    live "/products/new", ProductLive.New, :new
    live "/products/:id", ProductLive.Show
    live "/products/:id/edit", ProductLive.Edit, :edit
  end

Replace the contents or create the following files.

lib/mercury_web/live/product_live/index.ex:

defmodule MercuryWeb.ProductLive.Index do
  use MercuryWeb, :live_view

  alias Mercury.Products

  @impl true
  def mount(_params, _session, socket) do
    socket =
      socket
      |> assign(:page_title, "Listing Products")
      |> stream(:products, Products.list_products())

    {:ok, socket}
  end
end

lib/mercury_web/live/product_live/index.html.heex:

<.header>
  Listing Products
  <:actions>
    <.link navigate={~p"/products/new"}>
      <.button>New Product</.button>
    </.link>
  </:actions>
</.header>

<.table
  id="products"
  rows={@streams.products}
  row_click={fn {_id, product} -> JS.navigate(~p"/products/#{product}") end}
>
  <:col :let={{_id, product}} label="Name"><%= product.name %></:col>
  <:action :let={{_id, product}}>
    <div class="sr-only">
      <.link navigate={~p"/products/#{product}"}>Show</.link>
    </div>
    <.link navigate={~p"/products/#{product}/edit"}>Edit</.link>
  </:action>
</.table>

lib/mixquic_web/live/artisan_courses_live/new.ex:

defmodule MercuryWeb.ProductLive.New do
  use MercuryWeb, :live_view

  alias Mercury.Products.Product

  @impl true
  def mount(_params, _session, socket) do
    socket =
      socket
      |> assign(:page_title, "New Product")
      |> assign(:product, %Product{})

    {:ok, socket}
  end
end

lib/mercury_web/live/product_live/new.html.heex:

<.live_component
  module={MercuryWeb.ProductLive.FormComponent}
  id={:new}
  title={@page_title}
  action={@live_action}
  product={@product}
  navigate={~p"/products"}
/>

lib/mercury_web/live/product_live/edit.ex with this:

defmodule MercuryWeb.ProductLive.Edit do
  use MercuryWeb, :live_view

  alias Mercury.Products

  @impl true
  def mount(_params, _session, socket) do
    {:ok, socket}
  end

  @impl true
  def handle_params(%{"id" => id}, _url, socket) do
    {:noreply,
     socket
     |> assign(:page_title, "Edit Course")
     |> assign(:product, Products.get_product!(id))}
  end
end

lib/mixquic_web/live/artisan_courses_live/edit.html.heex with:

<.live_component
  module={MercuryWeb.ProductLive.FormComponent}
  id={@product.id}
  title={@page_title}
  action={@live_action}
  product={@product}
  navigate={~p"/products"}
/>

lib/mercury_web/live/product_live/show.ex:

defmodule MercuryWeb.ProductLive.Show do
  use MercuryWeb, :live_view

  alias Mercury.Products

  @impl true
  def mount(_params, _session, socket) do
    {:ok, socket}
  end

  @impl true
  def handle_params(%{"id" => id}, _, socket) do
    {:noreply,
     socket
     |> assign(:page_title, "Show course")
     |> assign(:product, Products.get_product!(id))}
  end
end

lib/mixquic_web/live/artisan_courses_live/show.html.heex:

<.header>
  Product <%= @product.id %>
  <:actions>
    <.link navigate={~p"/products/#{@product}/edit"} phx-click={JS.push_focus()}>
      <.button>Edit product</.button>
    </.link>
  </:actions>
</.header>

<.list>
  <:item title="Name"><%= @product.name %></:item>
</.list>
<div class="mt-4">
  <.label>Images</.label>
  <div class="grid justify-center md:grid-cols-2 lg:grid-cols-3 gap-5 lg:gap-7 my-10">
    <figure
      :for={image <- @product.images}
      class="rounded-lg border shadow-md max-w-xs md:max-w-none"
    >
      <img src={image} />
    </figure>
  </div>
</div>
<.back navigate={~p"/products"}>Back to products</.back>

lib/mercury_web/live/product_live/form_component.ex:

defmodule MercuryWeb.ProductLive.FormComponent do
  use MercuryWeb, :live_component

  alias Mercury.Products

  @impl true
  def render(assigns) do
    ~H"""
    <div>
      <.header>
        <%= @title %>
        <:subtitle>Use this form to manage product records in your database.</:subtitle>
      </.header>

      <.simple_form
        for={@form}
        id="product-form"
        phx-target={@myself}
        phx-change="validate"
        phx-submit="save"
      >
        <.input field={@form[:name]} type="text" label="Name" />
        <.label for="#images">Images</.label>
        <div id="images">
          <div
            class="p-4 border-2 border-dashed border-slate-300 rounded-md text-center text-slate-600"
            phx-drop-target={@uploads.images.ref}
          >
            <div class="flex flex-row items-center justify-center">
              <.live_file_input upload={@uploads.images} />
              <span class="font-semibold text-slate-500">or drag and drop here</span>
            </div>

            <div class="mt-4">
              <.error :for={err <- upload_errors(@uploads.images)}>
                <%= Phoenix.Naming.humanize(err) %>
              </.error>
            </div>

            <div class="mt-4 flex flex-row flex-wrap justify-start content-start items-start gap-2">
              <div
                :for={entry <- @uploads.images.entries}
                class="flex flex-col items-center justify-start space-y-1"
              >
                <div class="w-32 h-32 overflow-clip">
                  <.live_img_preview entry={entry} />
                </div>

                <div class="w-full">
                  <div class="mb-2 text-xs font-semibold inline-block text-slate-600">
                    <%= entry.progress %>%
                  </div>
                  <div class="flex h-2 overflow-hidden text-base bg-slate-200 rounded-lg mb-2">
                    <span style={"width: #{entry.progress}%"} class="shadow-md bg-slate-500"></span>
                  </div>
                  <.error :for={err <- upload_errors(@uploads.images, entry)}>
                    <%= Phoenix.Naming.humanize(err) %>
                  </.error>
                </div>
                <a phx-click="cancel" phx-target={@myself} phx-value-ref={entry.ref}>
                  <.icon name="hero-trash" />
                </a>
              </div>
            </div>
          </div>
        </div>
        <:actions>
          <.button phx-disable-with="Saving...">Save Product</.button>
        </:actions>
      </.simple_form>
    </div>
    """
  end

  @max_entries 3
  @max_file_size 5_000_000

  @impl true
  def update(%{product: product} = assigns, socket) do
    changeset = Products.change_product(product)

    {:ok,
     socket
     |> assign(assigns)
     |> assign_form(changeset)
     |> allow_upload(:images,
       accept: ~w(.png .jpg .jpeg),
       max_entries: @max_entries,
       max_file_size: @max_file_size
     )}
  end

  @impl true
  def handle_event("validate", %{"product" => product_params}, socket) do
    changeset =
      socket.assigns.product
      |> Products.change_product(product_params)
      |> Map.put(:action, :validate)

    {:noreply, assign_form(socket, changeset)}
  end

  def handle_event("save", %{"product" => product_params}, socket) do
    images =
      consume_uploaded_entries(socket, :images, fn meta, entry ->
        filename = "#{entry.uuid}#{Path.extname(entry.client_name)}"
        dest = Path.join(MercuryWeb.uploads_dir(), filename)

        File.cp!(meta.path, dest)

        {:ok, ~p"/uploads/#{filename}"}
      end)

    product_params = Map.put(product_params, "images", images)

    save_product(socket, socket.assigns.action, product_params)
  end

  def handle_event("cancel", %{"ref" => ref}, socket) do
    {:noreply, cancel_upload(socket, :images, ref)}
  end

  defp save_product(socket, :edit, product_params) do
    case Products.update_product(socket.assigns.product, product_params) do
      {:ok, _product} ->
        {:noreply,
         socket
         |> put_flash(:info, "Product updated successfully")
         |> push_navigate(to: socket.assigns.navigate)}

      {:error, %Ecto.Changeset{} = changeset} ->
        {:noreply, assign_form(socket, changeset)}
    end
  end

  defp save_product(socket, :new, product_params) do
    case Products.create_product(product_params) do
      {:ok, _product} ->
        {:noreply,
         socket
         |> put_flash(:info, "Product created successfully")
         |> push_navigate(to: socket.assigns.navigate)}

      {:error, %Ecto.Changeset{} = changeset} ->
        {:noreply, assign_form(socket, changeset)}
    end
  end

  defp assign_form(socket, %Ecto.Changeset{} = changeset) do
    assign(socket, :form, to_form(changeset))
  end
end

lib/mercury_web.ex:

  def static_paths, do: ~w(assets fonts images uploads favicon.ico robots.txt)
  def uploads_dir, do: Application.app_dir(:mercury, ["priv", "static", "uploads"])

lib/mercury/application.ex:

  @impl true
  def start(_type, _args) do
    MercuryWeb.uploads_dir() |> File.mkdir_p!()
    # ...
  end

Run migrations and start the server to try it:

mix ecto.migrate
mix phx.server

If you go now to http://localhost:4000/products

Create a new product:

Save it

and go to the show.html.heex:

We are now able to create a product, add images to it, and display them. So far so good.

(Find the code for this part on the official-live-file-input-docs branch of the git repo)

Editing the images of the product

Let's try to edit the product and change the images:

Now we see the issue. The existing images are not being shown the same way the product name is. If you select one image and save it, this will replace the three ones currently associated with the product. If you don't have the original images available to upload them again when editing the product, they will be replaced and lost.

That's not good. We need to:

1. show the existing images if they exist

2. be able to remove some of the existing images if we want to

3. add new images to the product and save it

4. ensure that we don't exceed the max number of images allowed.

Let's implement each of those points

Showing existing images

Let's show the images of the product. The images to show are the ones currently associated to the product, not the entries in the :images uploads map.

<.label for="#images">Images</.label>
<div id="images">
  <div class="flex flex-row flex-wrap justify-start items-start text-center text-slate-500 gap-2 mb-8">
    <div :for={image <- @product.images} class="border shadow-md pb-1">
      <figure class="w-32 h-32 overflow-clip">
        <img src={image} />
      </figure>
    </div>
  </div>

Now we can see the images in the edit form:

Removing existing images and adding new images

To remove the existing images from the product we need to take into consideration a couple of things. First, when we remove them, we won't show them on the edit form anymore, but we don't really apply the changes to the product until we click the Save Product button. We'll need to keep this in memory but not persist it until told so by the user.

We can use an removed_images assign to hold the images the user has removed so far. It will start as an empty list and, because it is the initial state, we'll assign it on the mount callback instead of the update callback. Let's add it:

  @impl true
  def mount(socket) do
    socket =
      socket
      |> assign(:removed_images, [])

    {:ok, socket}
  end

Now in the update handler we can calculate which images to show:

  @impl true
  def update(%{product: product} = assigns, socket) do
    changeset = Products.change_product(product)
    images_to_show = product.images -- socket.assigns.removed_images

    {:ok,
     socket
     |> assign(assigns)
     |> assign_form(changeset)
     |> assign(:images_to_show, images_to_show)
     |> allow_upload(:images,
       accept: ~w(.png .jpg .jpeg),
       max_entries: @max_entries,
       max_file_size: @max_file_size
     )}
  end

We add a new handler for the remove event, that will be sent each time we remove an existing image from the Product:

  @impl true
  def handle_event("remove", %{"image" => image}, socket) do
    product = socket.assigns.product
    removed_images = [image | socket.assigns.removed_images]
    images_to_show = product.images -- removed_images

    socket =
      socket
      |> assign(:removed_images, removed_images)
      |> assign(:images_to_show, images_to_show)

    {:noreply, socket}
  end

Let's tweak the template to correctly show the images and allow removing them:

<div :for={image <- @images_to_show} class="border shadow-md pb-1">
  <figure class="w-32 h-32 overflow-clip">
    <img src={image} />
  </figure>
  <a
    phx-click="remove"
    phx-target={@myself}
    phx-value-image={image}
    class="hover:text-slate-800"
  >
    <.icon name="hero-trash" />
  </a>
</div>

We can now remove images and if we reload without saving the changes, the original images are shown.

Let's modify the save event to correctly save the new set of images:

  def handle_event("save", %{"product" => product_params}, socket) do
    images_to_show = socket.assigns.images_to_show

    images =
      consume_uploaded_entries(socket, :images, fn meta, entry ->
        filename = "#{entry.uuid}#{Path.extname(entry.client_name)}"
        dest = Path.join(MercuryWeb.uploads_dir(), filename)

        File.cp!(meta.path, dest)

        {:ok, ~p"/uploads/#{filename}"}
      end)

    product_params = Map.put(product_params, "images", images_to_show ++ images)

    save_product(socket, socket.assigns.action, product_params)
  end

If we now remove an image and add a new one, the Product will correctly be saved

Nice, right?

Well, not completely. If you play around with the current code, you'll see that you can add more images than the maximum allowed by our requirements. In fact, you can go to the edit page and add 3 more images every time you save the form.

(Find the code for this part on the show-remove-add-images branch of the git repo)

Ensure we don't exceed the allowed number of images

Now we have reached the part where the problem is. As it currently is, the live_file_input has no way of knowing additional information other than the one you provide when you call the allow_update function. It is also not possible to modify the configuration after initialization unless you destroy it and start over. live_file_input doesn't know how many files to allow at any given moment because that depends on external information not available to it.

We can try some ways to address this issue.

Replacing the live_file_input configuration dynamically

We want the live_file_input component to allow only enough files so that they, when added to the images that remain in the product after any removal by the user, don't exceed the maximum number of allowed files. For example, if we have a restriction of 3 files maximum, the product has already 2 images associated, and the user removes 1 in the edit page, then the live_file_input component should allow 2 additional files to be uploaded.

This suggests we should be able to reconfigure the live_file_input with the correct value for :max_entries.

Something like this:

  @impl true
  def handle_event("remove", %{"image" => image}, socket) do
    product = socket.assigns.product
    removed_images = [image | socket.assigns.removed_images]
    images_to_show = product.images -- removed_images
    max_entries = @max_entries - length(images_to_show)

    socket =
      socket
      |> assign(:removed_images, removed_images)
      |> assign(:images_to_show, images_to_show)
      |> allow_upload(:images,
        accept: ~w(.png .jpg .jpeg),
        max_entries: max_entries,
        max_file_size: @max_file_size
      )

    {:noreply, socket}
  end

If we now go to the edit form and remove an image we see that, if we start with 3 images associated to the product and we remove 1, the live_file_input component allows us to upload only 1 file. If we remove 2, it will allow us to add 2 new files.

Trying to add 3 files, it shows the error message:

Removing one upload and trying to upload only two images, the error disappears.

It looks good!

Let's remove the last associated image and try to upload 3 completely new images:

Something went wrong.

  Parameters: %{"image" => "/uploads/4c8cb89e-a9ef-4bb6-8672-960570e79fce.jpg"}
[error] GenServer #PID<0.2173.0> terminating
** (ArgumentError) cannot allow_upload on an existing upload with active entries.

Use cancel_upload and/or consume_upload to handle the active entries before allowing a new upload.

    (phoenix_live_view 0.18.18) lib/phoenix_live_view/upload.ex:19: Phoenix.LiveView.Upload.allow_upload/3

There is a problem when we try to reconfigure the live_file_input if there are active entries (those are the files ready to be uploaded and that we are seeing the preview on the screen).

As the message states, one way to avoid this error is to not have active entries when removing images. We can check for active entries and cancel them before trying to reconfigure the live_file_input.

Phoenix has a method to do that. Let's use it:

  @impl true
  def handle_event("remove", %{"image" => image}, socket) do
    product = socket.assigns.product
    removed_images = [image | socket.assigns.removed_images]
    images_to_show = product.images -- removed_images
    max_entries = @max_entries - length(images_to_show)

    socket =
      socket
      |> assign(:removed_images, removed_images)
      |> assign(:images_to_show, images_to_show)
      |> maybe_cancel_uploads()
      |> allow_upload(:images,
        accept: ~w(.png .jpg .jpeg),
        max_entries: max_entries,
        max_file_size: @max_file_size
      )

    {:noreply, socket}
  end

  defp maybe_cancel_uploads(socket) do
    {socket, _} = Phoenix.LiveView.Upload.maybe_cancel_uploads(socket)
    socket
  end

This does the work, but not perfectly. When we cancel the active uploads in the remove handler, they disappear from our preview section.

Before removing the remaining associated image

After removing it, the previews of the two active uploads are gone. :(

We can mitigate this a little by showing an alert to the user informing her that the pending uploads will be removed.

<a
  phx-click="remove"
  phx-target={@myself}
  phx-value-image={image}
  class="hover:text-slate-800"
  data-confirm="Are you sure? The pending uploads will be removed and you need to select them again."
>
  <.icon name="hero-trash" />
</a>

Now a warning is shown when you try to remove an existing image and there are active uploads.

Still, I don't quite like this approach.

(Find the code for this part on the replace-live-file-input-config branch of the git repo)

Updating the max_entries option of live_file_input

Another way I tried was to change just the :max_entries option of the allow_update() call instead of completely replacing the configuration. Something like this:

  @impl true
  def handle_event("remove", %{"image" => image}, socket) do
    product = socket.assigns.product
    removed_images = [image | socket.assigns.removed_images]
    images_to_show = product.images -- removed_images
    max_entries = @max_entries - length(images_to_show)

    socket =
      socket
      |> assign(:removed_images, removed_images)
      |> assign(images_to_show: images_to_show)
      |> maybe_update_upload_config(max_entries)

    {:noreply, socket}
  end

  defp maybe_update_upload_config(socket, max_entries) do
    images_config = Map.get(socket.assigns.uploads, :images)

    new_uploads =
      Map.put(socket.assigns.uploads, :images, %{images_config | max_entries: max_entries})

    assign(socket, :uploads, new_uploads)
  end

But Phoenix wasn't happy about it, telling me that the :uploads is a reserved assign and I can't set it directly.

  Parameters: %{"image" => "/uploads/c121a0d2-2724-4c03-8f7f-dc7c78c07ccc.jpg"}
[error] GenServer #PID<0.2770.0> terminating
** (ArgumentError) :uploads is a reserved assign by LiveView and it cannot be set directly

A dead-end path.

(Find the code for this part on the update-live-file-input-config branch of the git repo)

Leveraging changeset validations

One way to completely work around the constraints of the live_file_input config is to leave the configuration as it is and, instead, rely on the changeset validations to apply the business logic checks.

Let's start with the Product's changeset. We can set a validation there to check the max amount of images allowed:

defmodule Mercury.Products.Product do
  use Ecto.Schema
  import Ecto.Changeset

  schema "products" do
    field :images, {:array, :string}
    field :name, :string

    timestamps()
  end

  @max_files 3

  @doc false
  def changeset(product, attrs) do
    product
    |> cast(attrs, [:name, :images])
    |> validate_required([:name, :images])
    |> validate_length(:images, max: @max_files, message: "Max number of images is #{@max_files}")
  end
end

The validate_length will check that we put at most @max_files elements on the images array.

Now we need to ensure the validate and save events put all the images the user wants to save in the images attribute and the changeset will do the rest.

Let's first create a helper function that does the validation of the current data in the socket: the @images_to_show and the entries in the live_file_input config:

  defp validate_images(socket, product_params, images_to_show) do
    images =
      socket.assigns.uploads.images.entries
      |> Enum.map(fn entry ->
        filename = "#{entry.uuid}#{Path.extname(entry.client_name)}"
        ~p"/uploads/#{filename}"
      end)

    product_params = Map.put(product_params, "images", images_to_show ++ images)

    changeset =
      socket.assigns.product
      |> Products.change_product(product_params)
      |> Map.put(:action, :validate)

    assign_form(socket, changeset)
  end

This helper doesn't really consume the entries, but does iterate over them to generate the filenames so that it can add them to the images that remain associated to the product. That's the new set of images that the user is trying to associate to the product. Then delegates to the Products.change_product/2 (that then calls the Product.changeset/2) to apply the validations. Finally it replaces the changeset in the socket so that the form in the browser shows the new errors if necessary.

With this in place the validate handler is now:

  @impl true
  def handle_event("validate", %{"product" => product_params}, socket) do
    images_to_show = socket.assigns.images_to_show

    socket =
      socket
      |> validate_images(product_params, images_to_show)

    {:noreply, socket}
  end

The save handler doesn't need changes as it already performs the same logic in addition to consuming the uploads.

There are other places where we need to do these checks, for example, when we remove an image from the product.

The remove handler is now this:

  @impl true
  def handle_event("remove", %{"image" => image}, socket) do
    product = socket.assigns.product
    removed_images = [image | socket.assigns.removed_images]
    images_to_show = product.images -- removed_images

    socket =
      socket
      |> assign(:removed_images, removed_images)
      |> assign(:images_to_show, images_to_show)
      |> validate_images(%{}, images_to_show)

    {:noreply, socket}
  end

In this case, as we are not validating the full form, we can just pass an empty product_params to the validate_images function so that the changeset can validate if the images are within the limits.

The cancel event, triggered when we remove one active upload from the form, becomes:

  def handle_event("cancel", %{"ref" => ref}, socket) do
    images_to_show = socket.assigns.images_to_show

    socket =
      socket
      |> cancel_upload(:images, ref)
      |> validate_images(%{}, images_to_show)

    {:noreply, socket}
  end

Here again, we're not validating the full form, and we can just pass an empty product_params to get the validations applied.

Finally, we show the error message in the template, if present:

<.label for="#images">Images</.label>

<.error :for={{msg, _} <- @form[:images].errors}><%= msg %></.error>

<div id="images">

With that in place, we can now try it.

We start with 3 images already in the Product:

If we try to upload an extra image, the validation shows the error:

We can now either remove one of the uploads or one of the images associated to the Product.

Removing one of the uploads:

Removing one of the associated images:

In both cases, the validations are reapplied and the error message is correctly handled.

I think this has better UX than the other options but still is not perfect. If you play around with the form you'll soon notice that the validations on other parts of the form are removed when we pass %{} as the product_params to the validate_images function. Sigh!

(Find the code for this part on the changeset-validations branch of the git repo)

Last resort

One final approach is to completely abandon live_file_input and changesets and perform all the logic in the form_component.ex module:

• Use a boolean assign to toggle the showing/hiding of the max files error message in the template

• Set/unset the toggle in each of the remove, cancel, validate, and save handlers

• Check the assign before saving so that we don't save the form if the toggle is set.

This will work and won't mess with the form's internal errors or with the live_file_input configuration.

But also it feels like is not in line with the "Elixir way" of doing things. But hey, it works, right?

Final considerations

This post is different than the ones I usually write, where I end with a happy feeling of accomplishment. This one feels like I personally couldn't find a way to make it work in an elegant way.

One thing is for sure, live_file_input is amazing for the use case it was created. I am sure it will evolve to support or at least facilitate other use cases like the one discussed here.

Judging by the lightning speed the Phoenix core team and contributors gift us with amazing tools, I'm sure minds more brilliant than mine will find a better way to handle scenarios like this one.

And just to avoid any misunderstanding, I'm not complaining about live_file_input, the bittersweet conclusions of this post fall totally on me and my limited experience with the way LiveView works.

What do you people think? Do you have another solution to this scenario that you want to share? I for sure want to learn it!

You can find the code for this article in this github repo

What's StatsD?

StatsD is a network daemon that listens for statistics, like counters and timers, sent over UDP or TCP and sends aggregates to one or more pluggable backend services.

StatsD does aggregation of data.

To visualize the data you need another tool. For example, to create graphs from the aggregated data you can use Graphite.

StatsD forwards aggregated data to

What's Graphite?

Graphite does two things:

1. Store numeric time-series data
2. Render graphs of this data on demand

In summary, StatsD collects and aggregates metrics and then sends them to Graphite to be rendered in nice graphs.

In this tutorial we'll send our Elixir metrics to StatsD and then we'll use the Graphite dashboard to see the graphs generated from our metrics.

Install StatsD and Graphite

There is a way to install separately StatsD server and Graphite but for this tutorial, I'll just use the official Graphite docker image that also includes the StatsD server.

In a terminal run:

docker run -d \
 --name graphite \
 --restart=always \
 -p 80:80 \
 -p 2003-2004:2003-2004 \
 -p 2023-2024:2023-2024 \
 -p 8125:8125/udp \
 -p 8126:8126 \
 graphiteapp/graphite-statsd

This will start the StatsD on port 8125 and the Graphite dashboard on port 80.

Let's add some sample data just to check that the graphs render correctly.

Evaluate this on another terminal:

echo "foo:2|c" | nc -u -w0 127.0.0.1 8125

Wait at least 10 seconds and then run this:

echo "foo:4|c" | nc -u -w0 127.0.0.1 8125

We have sent two counter values separated by at least 10 seconds.

Let's open a browser and go to http://localhost. You'll see something like this:

On the left panel expand the tree (Metrics -> stats_count) and select the foo entry.

You'll see the graph on the main panel populated with some data. But it is too difficult to see the data. Let's change the resolution of the graph.

Click on the clock icon:

And select a time range of 10 minutes. Now you can see clearly the two data points we send to StatsD:

Why we had to wait 10 seconds in the previous examples? It is because that's the default flush interval timeout that StatsD waits before aggregating the stats and sending them to Graphite.

Try sending this data one after the other without any delay:

echo "foo:3|c" | nc -u -w0 127.0.0.1 8125
echo "foo:5|c" | nc -u -w0 127.0.0.1 8125

If you go to the graph and click the Auto-Refresh button, you'll see that a new value appears on the graph, but the value is 8, that, as you can tell is the sum of 3 and 5 that we sent. StatsD aggregated all the values inside a flush interval and sent the result to Graphite.

Ok, enough with StatsD and Graphite. Let's do some Elixir.

Send Elixir metrics to StatsD

Let's continue using the project we created for this series. Clone it from here if you don't have it. We'll start working on the main branch

Add the :telemetry_metrics_statsd dependency to mix.exs:

  defp deps do
    [
      {:telemetry, "~> 1.0"},
      {:telemetry_metrics, "~> 0.6.1"},
      {:telemetry_metrics_statsd, "~> 0.6.0"}
    ]
  end

and run mix deps.get

Now edit the telemetry.ex file and add the TelemetryMetricsStatsd reporter to the list of children:

  def init(_arg) do
    children = [
      {TelemetryMetricsStatsd, metrics: metrics()}
      {Metrics.Telemetry.CustomReporter, metrics: metrics()}
    ]

    Supervisor.init(children, strategy: :one_for_one)
  end

Now our metrics will be also sent to the StatsD server. By default the TelemetryMetricsStatsd reporter sends the data to 127.0.0.1:8125, but you can configure the host and port passing additional :host and :port options.

One thing to know is that the reporter doesn't aggregate metrics. It just forwards them to StatsD and it is StatsD the one that will do the aggregation before sending the data to Graphite.

Let's generate some metrics. Start the application:

iex -S mix

and run this:

1..80 |> Enum.each(&Metrics.emit(&1))

Wait at least 10 seconds and then run:

1..100 |> Enum.each(&Metrics.emit(&1))

If you navigate the left pane and visualize the stats_count.metrics.emit.value you'll see a graph like the following, showing that the aggregated values in two subsequent intervals are 80 and 100. This is because we generated 80 metric events in the first flush interval and after 10 seconds we generated another 100 events in the second flush interval:

If you now visualize the stats.gauges.metrics.emit.value you'll see the sum metric values.

You'll see that in the first interval the value is 3240: the sum of the first 80 events.

In the second interval, the value is 8290: the sum of the next 100 events.

Now our application is using a new reporter to send the metrics to a StatsD server and we are using Graphite to visualize the metrics in configurable graphs.

There is no limit to the amount of data you can send to StatsD and visualize it with Graphite. You can also configure Graphite and save graphs arranged in dashboards so that you can see your data all together in a simple place

You can find the code for this article in the metrics-statsd branch in the github repo.

About

I'm Miguel Cobá. I write about Elixir, Elm, Software Development, and eBook writing.

• Follow me on Twitter
• Subscribe to my newsletter
• Read all my articles on my blog
• Get my books:
  • 100 Elixir Tips [FREE]
  • Deploying Elixir [FREE]
  • Deploying Elixir: Advanced Topics
  • eBook Writing Workflow for Developers

Elixir, Kubernetes, and minikube

in the Elixir Meetup #5 organized by the fine people on curiosum.com.

This talk is based on chapter 4 of my book "Deploying Elixir: Advanced Topics".

If you haven't gotten the book, this is an excellent way to see what the book is about.

Watch the recorded talk here:

About

I'm Miguel Cobá. I write about Elixir, Elm, Software Development, and eBook writing.

• Follow me on Twitter
• Subscribe to my newsletter
• Read all my articles on my blog
• Get my books:
  • 100 Elixir Tips [FREE]
  • Deploying Elixir [FREE]
  • Deploying Elixir: Advanced Topics
  • eBook Writing Workflow for Developers

Deploying Elixir: Advanced Topics

It is based on the feedback and suggestions I got from the readers of my previous book "Deploying Elixir".

They asked me to write more about clustering in Elixir and deploying to Kubernetes.

I'll take you from zero to deploying a Kubernetes cluster in the three big Cloud Providers.

In this book:

• I'll show you how to create an Elixir Release from your application and Dockerize it
• Learn what an Erlang cluster is and how to create one
• We explore the different strategies to create clusters using the libcluster library
• I'll show you how to deploy it to a Kubernetes cluster using minikube
• When you feel comfortable scaling your app I'll show you how to deploy to Amazon Web Service, Microsoft Azure, and Google Cloud Platform

You can download a sample chapter here:

Download sample chapter

If you prefer a video version, I presented part of chapter 4 in the this Elixir meetup talk:

Watch it here

Get the full book here:

Deploying Elixir: Advanced Topics

We talk about Elixir in polyglot environments (that is, where you have a lot of languages used at the same time), my beginnings in the software field, my books, and other sideprojects.

I had a lot of fun chatting with Sundi Myint and Owen Bickford who were amazing in guiding my ramblings to a coherent discussion. Thank you so much for having me.

You can listen to the podcast here:

Elixir Wizards S8E1

About

I'm Miguel Cobá. I write about Elixir, Elm, Software Development, and eBook writing.

• Follow me on Twitter
• Subscribe to my newsletter
• Read all my articles on my blog
• Get my books:
  • 100 Elixir Tips [FREE]
  • Deploying Elixir [FREE]
  • Deploying Elixir: Advanced Topics
  • eBook Writing Workflow for Developers

"Deploying Elixir: Advanced Topics"

What is it about?

I'll show you how to deploy to:

• AWS
• Azure
• GCP

You'll deploy like a PRO using:

• clustering

• Kubernetes orchestration

Tell me more

This book expands on my previous book "Deploying Elixir".

Many of you asked me to write about common scenarios used in the Enterprise! 🏭

Well, I did it! 💪

If you want to deploy your application like the PROS in the Industry do, this book is for you! 🎯

When it will be released?

The book will be published on May 3rd, 2022! 🗓️

But as a SPECIAL, TIME-LIMITED OFFER you can get 35% off the final price if you purchase it on PRESALE.

⚡ Get it on PRESALE and save 35% off the final price ⚡

First, let's recapitulate. In the previous article, I showed you how to create an Elixir Metrics Reporter to do the aggregation of your telemetry data and convert it to meaningful metrics. Now I am going to show you how to use the telemetry_metrics_prometheus library to expose this data so that Prometheus can retrieve it and process it.

Prerequisites

We are going to use the repo we did for the previous article. Clone it from here.

Install the telemetry_metrics_prometheus library

Add the telemetry_metrics_prometheus to your mix.exs dependencies.

  defp deps do
    [
      {:telemetry, "~> 1.0"},
      {:telemetry_metrics, "~> 0.6.1"},
      {:telemetry_metrics_prometheus, "~> 1.1.0"}
    ]
  end

Then get the new dependency:

mix deps.get

Configuring the Prometheus reporter

Now we need to add the reporter to telemetry.ex. Change the init/1 function to this:

  def init(_arg) do
    children = [
      {TelemetryMetricsPrometheus, metrics: metrics()},
      {Metrics.Telemetry.CustomReporter, metrics: metrics()}
    ]

    Supervisor.init(children, strategy: :one_for_one)
  end

And that should be all we need to do.

Unfortunately, it is not. The current version of the library doesn't allow several metrics to have the same metric name. Right now, we have this:

  defp metrics do
    [
      counter("metrics.emit.value"),
      sum("metrics.emit.value")
    ]
  end

We need to use different metric names but still handle the same events. We can take advantage of the second parameter to the metric functions that allow us to be explicit about events and measurements. Let's change it to this:

  defp metrics do
    [
      counter("metrics.emit.value.counter", event_name: [:metrics, :emit], measurement: :value),
      sum("metrics.emit.value.sum", event_name: [:metrics, :emit], measurement: :value)
    ]
  end

You can see that now the metric names are different but the event names and measurements are correct. With this, telemetry_metrics_prometheus will work correctly.

Testing the metrics generated

The Prometheus library includes a web server that exposes a /metrics endpoint on port 9568. We can use it to check the metrics collected by Prometheus. Let's try it. Start your app with iex:

iex -S mix

and go to http://localhost:9568/metrics

And you'll see an empty page. Disappointing. But that's because we haven't yet generated any event.

Let's emit some events. In your just opened iex session write this:

iex -S mix
iex(1)> 1..100 |> Enum.each(&Metrics.emit(&1))
Metric: Elixir.Telemetry.Metrics.Counter. Current value: {1, 0}
Metric: Elixir.Telemetry.Metrics.Sum. Current value: {1, 1}
...
Metric: Elixir.Telemetry.Metrics.Counter. Current value: {100, 4950}
Metric: Elixir.Telemetry.Metrics.Sum. Current value: {100, 5050}
:ok
iex(2)>

As you can see, I am emitting 100 events, with values from 1 to 100. As we have count and a sum metrics defined, we should get the count of events (100) and the sum of all those events (the sum from values from 1 to 100 is 5050).

From the iex output, at least we can affirm that our CustomReporter works. What about the Prometheus reporter?

Go again to http://localhost:9568/metrics and you'll see this:

Yay, the output of the /metrics endpoint is correctly counting and adding the values we emit with our events. Good.

We now need Prometheus to ingest this information. So far, our app is only exposing that endpoint but if nobody access it, is not really useful.

Install Prometheus

Let's install Prometheus. You can go to the prometheus page and follow the install instructions. Or, if you are in mac, use brew to install it.

For this tutorial, I am assuming you used brew.

brew install prometheus

This will install it locally and you can either start it manually or as a service. But first, we need to configure it. The output of brew install will show where the configuration files are. You should see messages similar to these:

==> Caveats
When run from `brew services`, `prometheus` is run from
`prometheus_brew_services` and uses the flags in:
   /usr/local/etc/prometheus.args

To restart prometheus after an upgrade:
  brew services restart prometheus
Or, if you don't want/need a background service you can just run:
  /usr/local/opt/prometheus/bin/prometheus_brew_services

Be sure to look at that output to know where it is. On my laptop, the config file is in /usr/local/etc/prometheus.yml

Open it and add a new section for our elixir app:

global:
  scrape_interval: 15s

scrape_configs:
  - job_name: "prometheus"
    static_configs:
    - targets: ["localhost:9090"]
  - job_name: "telemetry_metrics_prometheus"
    static_configs:
    - targets: ["localhost:9568"]

I only added the last job_name named "telemetry_metrics_prometheus". You can see that I specified the 9568 port and left out the /metrics part. Prometheus by default tries to access the /metrics endpoint of the target.

Save it and now we can start the Prometheus server:

brew services restart prometheus
Stopping `prometheus`... (might take a while)
==> Successfully stopped `prometheus` (label: homebrew.mxcl.prometheus)
==> Successfully started `prometheus` (label: homebrew.mxcl.prometheus)

Prometheus runs on port 9090 so open http://localhost:9090/ and you'll see something like this:

Not much to see yet, but that's because we haven't specified what metric we want to see. If you remember the metrics we expose on /metrics have the names metrics_emit_value_sum and metrics_emit_value_counter. Let's use metrics_emit_value_sum to see the values Prometheus scrapped from our Elixir app. Put it in the "Expression" input field and click on the "Execute" button.

You'll see now the last value Prometheus scrapped from our Elixir app:

Nice. If you go to the graph tab you'll see the value too.

As we only have a single data point, we see a simple horizontal line. Let's add more values, but let's wait 15 seconds between each one (that's the interval Prometheus waits between scraps).

1..80 |> Enum.each(&Metrics.emit(&1))
1..110 |> Enum.each(&Metrics.emit(&1))
1..50 |> Enum.each(&Metrics.emit(&1))
1..170 |> Enum.each(&Metrics.emit(&1))

After that, if you refresh the graph, you'll see something like this:

That's it. Prometheus is now scraping the values from the /metrics endpoint in our Elixir application.

What's next

There is a lot more to explore from here. You could for example integrate with Grafana to create stunning dashboards for your metrics. If you go that way, don't forget to take a look at the prom_ex library, because it automates exposing your Prometheus data and creating Grafana dashboards on application start.

Summary

We learned:

• what the telemetry_metrics_prometheus library is
• how to scrap our metrics with Prometheus

Source code

You can find the source code for this article in this GitHub repository. Check the metrics-prometheusbranch.

About

I'm Miguel Cobá. I write about Elixir, Elm, Software Development, and eBook writing.

• Follow me on Twitter
• Subscribe to my newsletter
• Read all my articles on my blog
• Get my books:
  • 100 Elixir Tips [FREE]
  • Deploying Elixir [FREE]
  • Deploying Elixir: Advanced Topics
  • eBook Writing Workflow for Developers

• generate events
• create handler functions that will receive the events
• register which handlers will be called when an event occurs

The principal advantage of telemetry is a standardized way to emit events and collect measurements in Elixir. This way, third-party libraries and code that we write ourselves can use it to generate events in a consistent way.

Let's now take a step back and consider a couple of things. An event is just a measure of something that happened at a given point in time. By itself doesn't tell us much. We might need more data to compare it against or to infer some trend or some rate of change. But the measure by itself is not very useful.

With that in mind, the telemetry developers created a library called telemetry_metrics. According to their documentation, telemetry_metrics is:

Common interface for defining metrics based on :telemetry events.

Ok, but what are metrics, and how it is different from measurements taken with events? Well, again according to their docs:

Metrics are aggregations of Telemetry events with specific name, providing a view of the system's behavior over time.

Ok, it seems clear, right? Not really.

If you start using the telemetry_metrics library you'll notice that in fact DOESN'T AGGREGATE anything!!!

What?

Exactly that. Doesn't aggregate anything.

If you generate 1000 events with a measurement each, this library is not going to keep, manipulate, calculate, summarize or do anything by itself.

So, what is useful for, you might ask?

Its aim is to define what kind of metrics you're going to create from events. That's it. Just a declaration, a definition, a contract if you will. Nothing else.

So, how do you get the real metrics, the processed results of your carefully collected measurements?

For that, you need something else. You need a reporter.

The reporter's responsibility is to aggregate and process each event received so that it conforms to the specified metric it handles.

telemetry_metrics ships with a simple reporter called ConsoleReporter that doesn't do much. It only outputs the measurement and metric details to the console. Not much, but allows you to verify that everything works and you're collecting data to generate metrics correctly.

For more useful reporters you should either write them yourself or get them from someone that has already written it.

In this article, I am going to show you how to use the default ConsoleReporter and how to create a custom reporter.

Prerequisites

Use asdf and install these versions of elixir and erlang:

asdf install erlang 24.2.1
asdf global erlang 24.2.1
asdf install elixir 1.13.3-otp-24
asdf global elixir 1.13.3-otp-24

Create an Elixir app

mix new metrics --sup

This will create an Elixir app with an application callback where we can attach our telemetry events and configure our metrics.

Install dependencies

Add telemetry and telemetry_metrics to your mix.exs:

  defp deps do
    [
      {:telemetry, "~> 1.0"},
      {:telemetry_metrics, "~> 0.6.1"}
    ]
  end

Create an event emitter

Let's create a simple event emitter that we can use to test our metrics. Open the lib/metrics.ex file that mix created for you and replace its contents with this:

defmodule Metrics do
  def emit(value) do
    :telemetry.execute([:metrics, :emit], %{value: value})
  end
end

As you can see it uses :telemetry.execute to emit an event passing the value we provide.

Define some metrics

Ok, now the interesting part: let's define some metrics we want to collect. telemetry_metrics allows to create several types of metrics:

• counter/2 which counts the total number of emitted events
• sum/2 which keeps track of the sum of selected measurement
• last_value/2 holding the value of the selected measurement from the most recent event
• summary/2 calculating statistics of the selected measurement, like maximum, mean, percentiles, etc.
• distribution/2 which builds a histogram of selected measurement

We are going to start with the basics and define a counter metric, assuming we want to count how many times the event has happened.

Create a new file lib/metrics/telemetry.ex and put this in it:

defmodule Metrics.Telemetry do
  use Supervisor
  import Telemetry.Metrics

  def start_link(arg) do
    Supervisor.start_link(__MODULE__, arg, name: __MODULE__)
  end

  def init(_arg) do
    children = [
      {Telemetry.Metrics.ConsoleReporter, metrics: metrics()}
    ]

    Supervisor.init(children, strategy: :one_for_one)
  end

  defp metrics do
    [
      counter("metrics.emit.value")
    ]
  end
end

Let's analyze it. First of all, it is a simple Supervisor that we need to start from some other process in order to work. We'll attach it to our main application supervisor tree. More on that later.

Then, the init function is configuring the children for this supervisor. We have a single child here, the ConsoleReporter. And we are passing a list of metrics to it as initial arguments.

Here is where it becomes interesting. Telemetry.Metrics.counter is the definition of the metric we want to use. We pass a string: "metrics.emit.value". This string will be split using the period as separator and everything but the last part (metrics.emit), will be the event to attach to. The last part (value) will be taken from the measurement passed to the telemetry event handler.

Let's explain it. We have this call in Metrics.emit/1:

    :telemetry.execute([:metrics, :emit], %{value: value})

and we have a metrics definition in Metrics.Telemetry.metrics/0:

      counter("metrics.emit.value")

As you see, the counter is going to attach a handler for the [:metrics, :emit] event and get the :value attribute from the second argument to telemetry.execute call (the measurement):

• event: "metrics.emit" -> [:metrics, :emit]
• measurement attribute: "value" -> :value

If we were collecting the :query_time of [:my_app, :repo, :query] event, we would write "my_app.repo.query.query_time".

Ok, let's continue.

We are using the ConsoleReporter to collect our metrics. We defined the counter metric. We also said that ConsoleReporter do nothing but output the values received. That's is enough, for now, to check that we are in fact collecting metrics with our ConsoleReporter.

But one thing is missing. We need to attach our Metrics.Telemetry supervisor to the application supervisor, otherwise, nothing will happen. Open application.ex and change the start/2 function to this:

  def start(_type, _args) do
    children = [
      Metrics.Telemetry
    ]

    opts = [strategy: :one_for_one, name: Metrics.Supervisor]
    Supervisor.start_link(children, opts)
  end

As you see we are putting our Metrics.Telemetry as a child of the application Supervisor.

Test it

Let's try it with iex. Open a shell terminal, get the dependencies and open an iex session:

mix deps.get
iex -S mix

Emit an event:

iex(1)> Metrics.emit(4)
[Telemetry.Metrics.ConsoleReporter] Got new event!
Event name: metrics.emit
All measurements: %{value: 4}
All metadata: %{}

Metric measurement: :value (counter)
Tag values: %{}

:ok
iex(2)>

Yay, it works. Although we didn't attach any handler to the [:metrics, :emit] event manually, the ConsoleReporter handled it for us thanks to telemetry_metrics. As I said, it only echoes what is passed in, but the important thing here is that it works.

If the reporter were a little more advanced it would do something with the passed value.

CustomReporter

Let's create a simple CustomReporter by ourselves to see how this works. One thing to notice is that the reporter needs to have a memory of previous values in order to do its calculations. In our case, we need to remember how many events we have received in order to increment it every time we receive a new event. We could use :ets or a GenServer or Agent depending on how complex our implementation needs to be. For this tutorial, I am going to use an Agent as it is very simple to use to keep some persistent value.

Create a file called lib/metrics/telemetry/reporter_state.ex and write this:

defmodule Metrics.Telemetry.ReporterState do
  use Agent

  def start_link(initial_value) do
    Agent.start_link(fn -> initial_value end, name: __MODULE__)
  end

  def value do
    Agent.get(__MODULE__, & &1)
  end

  def increment do
    Agent.update(__MODULE__, &(&1 + 1))
  end
end

Let's attach it as a child to our main application supervisor so that it is started when the app starts. Edit application.ex and change the start/2 function to this:

  def start(_type, _args) do
    children = [
      {Metrics.Telemetry.ReporterState, 0},
      Metrics.Telemetry
    ]

    opts = [strategy: :one_for_one, name: Metrics.Supervisor]
    Supervisor.start_link(children, opts)
  end
end

Now the application will automatically start a counter agent with an initial value of 0 that we can use from every other part of the app by using its name Metrics.Telemetry.CounterAgent. Let's try it:

iex -S mix
iex(1)> Metrics.Telemetry.ReporterState.value()
0
iex(2)> Metrics.Telemetry.ReporterState.increment()
:ok
iex(3)> Metrics.Telemetry.ReporterState.value()    
1

Nice, we have our agent working. We can use it to maintain the state of our custom reporter.

Let's write our reporter. Create a file called lib/metrics/telemetry/custom_reporter.ex:

defmodule Metrics.Telemetry.CustomReporter do
  use GenServer

  alias Metrics.Telemetry.ReporterState
  alias Telemetry.Metrics

  def start_link(metrics: metrics) do
    GenServer.start_link(__MODULE__, metrics)
  end

  @impl true
  def init(metrics) do
    Process.flag(:trap_exit, true)

    groups = Enum.group_by(metrics, & &1.event_name)

    for {event, metrics} <- groups do
      id = {__MODULE__, event, self()}
      :telemetry.attach(id, event, &__MODULE__.handle_event/4, metrics)
    end

    {:ok, Map.keys(groups)}
  end

  def handle_event(_event_name, measurements, metadata, metrics) do
    metrics
    |> Enum.map(&handle_metric(&1, measurements, metadata))
  end

  defp handle_metric(%Metrics.Counter{} = metric, _measurements, _metadata) do
    ReporterState.increment()

    current_value = ReporterState.value()

    IO.puts("Metric: #{metric.__struct__}. Current value: #{current_value}")
  end

  defp handle_metric(metric, _measurements, _metadata) do
    IO.puts("Unsupported metric: #{metric.__struct__}")
  end

  @impl true
  def terminate(_, events) do
    for event <- events do
      :telemetry.detach({__MODULE__, event, self()})
    end

    :ok
  end
end

A lot of things happening here, but don't get distracted by those. What you should notice is the handle_event function that distributes the event to all the specific metric handlers to, well, handle the event.

The handle_metric function uses ReporterState to keep the current state and increments the running counter that we use to track how many times we have received this event. It also prints the new current state to the console.

Let's use our CustomReporter instead of the ConsoleReporter. Change the init/1 function in telemetry.ex to be like this:

  def init(_arg) do
    children = [
      {Metrics.Telemetry.CustomReporter, metrics: metrics()}
    ]

    Supervisor.init(children, strategy: :one_for_one)
  end

As you see, now we use our CustomReporter and we pass the metrics to it just as we did with the ConsoleReporter. Let's try it. Open the console and start iex:

iex -S mix
iex(1)> Metrics.emit(4)
Metric: Elixir.Telemetry.Metrics.Counter. Current value: 1
:ok
iex(2)> Metrics.emit(5)
Metric: Elixir.Telemetry.Metrics.Counter. Current value: 2
:ok
iex(3)> Metrics.emit(2)
Metric: Elixir.Telemetry.Metrics.Counter. Current value: 3
:ok

And there it is, now the counter is correctly keeping track of how many times we have received the event.

Let's review a few more things about the CustomReporter. Most of them are better explained in the official docs here.

It traps the exit signal to detach the handlers when the app terminates or the process is about to finish. It also groups together similar events that require several metrics tracking. For example, suppose you want to collect counter and sum metrics for the event "my_app.repo.query.query_time". This code gets the event once and then calls handle_metric two times, one with the %Telemetry.Metrics.Counter{} as the first argument and one with %Telemetry.Metrics.Sum{} as the first argument. And for both of those calls, the second argument is the measurement we just received.

Add support for more metrics

Ok, so far so good. But if you notice we are just handling one metric. Let's add support for the sum metric.

First, to sum all the event measurements, we need to also keep track of the running sum so far. Our ReporterState is only handling a single integer as the state. Let's change it to now store a tuple, with the first element being the count and the second element being the sum.

Change ReporterState to this:

defmodule Metrics.Telemetry.ReporterState do
  use Agent

  def start_link(initial_value) do
    Agent.start_link(fn -> initial_value end, name: __MODULE__)
  end

  def value do
    Agent.get(__MODULE__, & &1)
  end

  def increment do
    Agent.update(__MODULE__, fn {count, sum} -> {count + 1, sum} end)
  end

  def sum(value) do
    Agent.update(__MODULE__, fn {count, sum} -> {count, sum + value} end)
  end
end

Now the agent has a composite state and a function to increment the count part and a function to add to the total part. This is a naive implementation, of course, and doesn't even care about both parts of the state getting out of sync, but for our purposes, it will suffice. We needed a way to store state, we have it.

Let's change our CustomReporter. Remove the old handle_metric functions and put these ones instead:

  defp handle_metric(%Metrics.Counter{} = metric, _measurements, _metadata) do
    ReporterState.increment()

    current_value = ReporterState.value()

    IO.puts("Metric: #{metric.__struct__}. Current value: #{inspect(current_value)}")
  end

  defp handle_metric(%Metrics.Sum{} = metric, %{value: value}, _metadata) do
    ReporterState.sum(value)

    current_value = ReporterState.value()

    IO.puts("Metric: #{metric.__struct__}. Current value: #{inspect(current_value)}")
  end

  defp handle_metric(metric, _measurements, _metadata) do
    IO.puts("Unsupported metric: #{metric.__struct__}")
  end

Again, nothing fancy. Now we handle a second type of metric, the %Metrics.Sum{} and, similarly to the count one, we use the ReporterState to keep track of the event measurements so far.

Let's tell telemetry_metrics to also handle this type of metrics. Update the metrics/0 function of telemetry.ex to this:

  defp metrics do
    [
      counter("metrics.emit.value"),
      sum("metrics.emit.value")
    ]
  end

Now the sum metric is also being collected.

Finally, change the start/2 function in application.ex to this:

 def start(_type, _args) do
    children = [
      {Metrics.Telemetry.ReporterState, {0, 0}},
      Metrics.Telemetry
    ]

    opts = [strategy: :one_for_one, name: Metrics.Supervisor]
    Supervisor.start_link(children, opts)
  end

We are set. Let's try it. In the shell:

iex -S mix
iex(1)> Metrics.emit(4)
Metric: Elixir.Telemetry.Metrics.Counter. Current value: {1, 0}
Metric: Elixir.Telemetry.Metrics.Sum. Current value: {1, 4}
:ok
iex(2)> Metrics.emit(3)
Metric: Elixir.Telemetry.Metrics.Counter. Current value: {2, 4}
Metric: Elixir.Telemetry.Metrics.Sum. Current value: {2, 7}
:ok
iex(3)> Metrics.emit(2)
Metric: Elixir.Telemetry.Metrics.Counter. Current value: {3, 7}
Metric: Elixir.Telemetry.Metrics.Sum. Current value: {3, 9}
:ok
iex(4)> Metrics.emit(1)
Metric: Elixir.Telemetry.Metrics.Counter. Current value: {4, 9}
Metric: Elixir.Telemetry.Metrics.Sum. Current value: {4, 10}
:ok

And that's it. Our CustomReporter is now capable of tracking two different metrics for us. Of course, a production system will have a better way to store and keep track of the series of measurements so that it has more guarantees about the data ingested. But that is out of the scope of this article.

Summary

We learned that telemetry_metrics:

• offers a way to define 5 types of metrics in a standard way.
• doesn't really care or dictates how the measurements should be stored, aggregated, or manipulated
• the reporter responsibility is to ingest and implement the real manipulation of the data

There are several open-source implementations of reporters that allow us to make our measurements available to tools like Prometheus or StatsD servers.

In a future article, I'll talk about integrating with them.

Source code

You can find the source code for this article in this github repository.

About

I'm Miguel Cobá. I write about Elixir, Elm, Software Development, and eBook writing.

• Follow me on Twitter
• Subscribe to my newsletter
• Read all my articles on my blog
• Get my books:
  • 100 Elixir Tips [FREE]
  • Deploying Elixir [FREE]
  • Deploying Elixir: Advanced Topics
  • eBook Writing Workflow for Developers

In this article I am going to show you what Telemetry is, why is needed, and how to use it in your own Elixir project.

Software Telemetry

First of all, what is telemetry?

Wikipedia explains it very clearly:

Telemetry is the [...] collection of measurements [...] and their automatic transmission [...] for monitoring

We can apply that to software systems, too, by collecting measurements while a software system is running and keeping track of that data.

Why would we do this, you may ask? Glad you ask. There are several reasons, but the main one is to discover what is happening inside our systems and then optimize them based on our findings.

What problem does telemetry solve for us? It gives us visibility into things that are important to us to measure and monitor.

Maybe we want to keep track of CPU utilization to see why our AWS bill is so high, or we need to see how many times a query is executed and how long it took to finish in our PostgreSQL database.

Measuring things and monitoring is not enough. If we don't use the collected data to make decisions and change or optimize the system measured then we might as well not measure anything at all.

By measuring our systems we get several benefits:

• we can look for parts of the system that are slow and can be improved.
• we could discover parts of the system that are not used frequently enough and that maybe can be scaled down to reduce our infrastructure bill.
• we could aggregate how many resources a customer is using every month and generate utilization graphs and other insights to make better use of our system. This would allow us to invite the customer to upgrade to a higher plan that includes this kind of data analysis.

In summary, software telemetry allows us to measure and act upon the collected data of a running system for our own benefit.

Telemetry in Elixir

There are several ways to take measures from a running Elixir app.

For example, the following fragment would take the duration of a function execution:

def some_function do
    start = System.monotonic_time()
    # do something that you want to measure
    stop = System.monotonic_time()
    record_measure("operation took: ", stop - start)
    # ...
end

In the strict sense, that is telemetry. But is not the best approach. It has several drawbacks:

• It is repetitive
• You have to reimplement it in every app you want to use it
• You have no way to disable measuring, for example, during unit tests
• more things that I can't think right now

Elixir Telemetry library

Given that telemetry is something that many people want, a library was created to address this need. It is called telemetry.

Advantages:

• provides a standard interface to measure and emit telemetry data
• it is small and simple to use

How does it work? It is very simple:

• you execute a measurement
• you create a function that will be invoked when the measure is taken
• you attach the function to the event measured

Using telemetry

Let's try a simple example to see how easy is to start collecting telemetry data.

I am going to create a very simple Elixir for a fictional grocery store. We want to measure each sale.

First, create a simple Elixir project:

mix new telemetry_intro
cd telemetry_intro

Add telemetry to the list of dependencies in mix.exs:

defp deps() do
  [
    {:telemetry, "~> 1.0"}
  ]
end

Download the dependencies:

mix deps.get

Now our business logic module. Create a file named lib/grocery/store.ex and put this inside:

defmodule TelemetryIntro.Grocery.Store do
  def sale(product, quantity, amount) do
    total = quantity * amount

    "Sold #{product}: #{quantity} units at #{amount} each. Total #{total}"
  end
end

As you see it is very simple. It just receives some parameters, does something, and then returns a value. Doesn't have any measure code yet.

Let's just try it with iex. In the shell, write:

iex -S mix
iex(1)> TelemetryIntro.Grocery.Store.sale("apple", 4, 0.5)
"Sold apple: 4 units at 0.5 each. Total 2.0"

Nothing unexpected here. Let's add some measurements. Say we want to keep track of the total and also to track what product was sold.

Change the sale function to be like this:

defmodule TelemetryIntro.Grocery.Store do
  def sale(product, quantity, amount) do
    total = quantity * amount

    :telemetry.execute(
      [:grocery, :store, :sale],
      %{total: total},
      %{product: product}
    )

    "Sold #{product}: #{quantity} units at #{amount} each. Total #{total}"
  end
end

Good. Now we are doing something. We are using the execute function of the telemetry module and we are passing three things:

• the event name as a list of keywords
• the measurement we are interested in (sale total in our example)
• additional metadata we might need (here I am just putting the product name)

The metadata could be anything we want: our users' IP address, some tracking header from a web request, the user id, etc.

The measurement is what we are interested in. This is what we'll collect and analyze and then use to decide things. It can be anything, but it must be a measurement of something.

Finally is the event name. You can put anything here. Most people put a hierarchy of words to easily identify the events. What is important to keep in mind is that this event name will be used later to match which function will handle this event.

Let's see how the counterpart, the event handler, looks.

Create a file lib/metrics/instrumenter.ex

defmodule TelemetryIntro.Metrics.Instrumenter do
  require Logger

  def handle_event([:grocery, :store, :sale], measurements, metadata, _config) do
    Logger.info("[Sale telemetry: #{measurements.total}] total for #{metadata.product}")
  end
end

It is very simple. It has a function (the name doesn't matter but the convention is to name it handle_event) that has as the first parameter the keyword list we are going to match against. Then the measurements and metadata. The last parameter doesn't matter right now. Just ignore it.

This function is going to be called every time the execute function is called and the event name matches. That means that this function must avoid blocking execution for much time as it will be evaluated synchronously.

Finally, we do a lame thing and just pass the received info to Logger.info. Ideally, we'll send that telemetry data to some repository for further processing and analysis but that is outside of the scope of this article (I'll cover it in a future one, tho).

What you need to notice here is that we decoupled the emitting of the telemetry data with the handling of the data collected. Our app can use :telemetry.execute everywhere with the same standard interface and, as long as we have a corresponding function to handle that event, we'll be collecting that data.

Even more important, if we don't have an event handler for that event, our emitting code will still work. It won't even notice there is nothing collecting data.

We are only missing one thing. We need to specify which function will handle a given event. We use the :telemetry.attach function to do that.

Create a function called setup in the Instrumenter module:

defmodule TelemetryIntro.Metrics.Instrumenter do
  require Logger

  def setup do
    events = [
      [:grocery, :store, :sale]
    ]

    :telemetry.attach_many("telemetry-intro-instrumenter", events, &__MODULE__.handle_event/4, nil)
  end

  def handle_event([:grocery, :store, :sale], measurements, metadata, _config) do
    Logger.info("[Sale telemetry: #{measurements.total}] total for #{metadata.product}")
  end
end

As you can see, it is passing to :telemetry.attach_many a list of event names to monitor and the function that will handle all those event names.

By using pattern matching, the same handle_event function can handle all the necessary events in your app.

Now we need to call this setup function when the app starts.

Let's change the way our elixir app starts by modifying the mix.exs file and changing the application function inside it so that it is like this:

  def application do
    [
      extra_applications: [:logger],
      mod: {TelemetryIntro, []}
    ]
  end

We added the mod: to specify a module that will be called when the application starts to execute some custom code. Our custom code will be the telemetry configuration.

Now replace the contents of the lib/telemetry_intro.ex module with this:

defmodule TelemetryIntro do
  use Application

  def start(_type, _args) do
    TelemetryIntro.Metrics.Instrumenter.setup()

    children = []
    Supervisor.start_link(children, strategy: :one_for_one)
  end
end

This module implements the start callback of the Application behavior. This function's responsibility is to spawn and link a supervisor and return {:ok, pid} or {:ok, pid, state}. Here we just start a supervisor with no children. That will suffice for this example.

Let's try it. Start again the application with iex:

iex -S mix
iex(1)> TelemetryIntro.Grocery.Store.sale("apple", 4, 0.2)

00:18:01.867 [info]  [Sale telemetry: 0.8] total for apple
"Sold apple: 4 units at 0.2 each. Total 0.8"

Great, it works. The first output line is the result of handle_event call. The second one is the return value of the sale function.

Let's see what happens if we don't configure a handler for the event.

Comment the TelemetryIntro.Metrics.Instrumenter.setup() in telemetry_intro.ex

defmodule TelemetryIntro do
  use Application

  def start(_type, _args) do
    # TelemetryIntro.Metrics.Instrumenter.setup()

    children = []
    Supervisor.start_link(children, strategy: :one_for_one)
  end
end

And try again with iex:

iex -S mix
iex(1)> TelemetryIntro.Grocery.Store.sale("apple", 4, 0.2)
"Sold apple: 4 units at 0.2 each. Total 0.8"

As you can see, the code still works, even if there is no handler configured for that event. Our app still calls the :telemetry.execute function but nothing breaks when nothing is configured to handle that event. How cool is that?

We learned how to emit events and handle them. You can send this info to a third-party error monitoring service or ingest it to a big data store. From here, the sky is the limit.

Summary

In this article we learned:

• what is telemetry
• how to add it to an Elixir project
• how to measure and collect data
• how telemetry works

As we have seen, telemetry is a powerful tool to inspect the behavior of a running system.

As long as we use the collected data for making decisions to improve our system or its behavior, data collection through telemetry is an investment that every system must seriously consider making.

If you want to learn more about telemetry, you can read the full documentation here

About

I'm Miguel Cobá. I write about Elixir, Elm, Software Development, and eBook writing.

• Follow me on Twitter
• Subscribe to my newsletter
• Read all my articles on my blog
• Get my books:
  • 100 Elixir Tips [FREE]
  • Deploying Elixir [FREE]
  • Deploying Elixir: Advanced Topics
  • eBook Writing Workflow for Developers

• 100 knowledge pills for you
• 100 curated examples created for you
• 100 ways to master Elixir
• 100 opportunities to increase your knowledge

IT IS FREE TO DOWNLOAD!!!

Download it now

About

I'm Miguel Cobá. I write about Elixir, Elm, Software Development, and eBook writing.

• Follow me on Twitter
• Subscribe to my newsletter
• Read all my articles on my blog
• Get my books:
  • 100 Elixir Tips [FREE]
  • Deploying Elixir [FREE]
  • Deploying Elixir: Advanced Topics
  • eBook Writing Workflow for Developers

As the Twitter search interface isn't good I decided to compile them in an ebook so that you can download it and have it as a reference that is easy to search.

It contains:

• 75 tips already published in my Twitter account
• 25 new, unpublished tips

All of them were carefully curated, edited, and selected by me to show the expressive power and elegance of the Elixir programming language.

The best part: it is FREE!!!

You can get it in pre-order now and you'll receive it in your email inbox on January 5th, 2022.

Pre-order it now here 👇🏼

100 Elixir Tips

About

I'm Miguel Cobá. I write about Elixir, Elm, Software Development, and eBook writing.

• Follow me on Twitter
• Subscribe to my newsletter
• Read all my articles on my blog
• Get my books:
  • 100 Elixir Tips [FREE]
  • Deploying Elixir [FREE]
  • Deploying Elixir: Advanced Topics
  • eBook Writing Workflow for Developers

We talked about deployment in the Elixir world.

I was super nervous and really suffering an episode of imposter syndrome, but I think in the end it was not so bad.

You can listen to the discussion here

About

I'm Miguel Cobá. I write about Elixir, Elm, Software Development, and eBook writing.

• Follow me on Twitter
• Subscribe to my newsletter
• Read all my articles on my blog
• Get my books:
  • 100 Elixir Tips [FREE]
  • Deploying Elixir [FREE]
  • Deploying Elixir: Advanced Topics
  • eBook Writing Workflow for Developers

• All together in a single place
• Nicely formatted
• Searchable
• With improved wording
• With better source code highlight

But I have also added a NEW, NEVER PUBLISHED BEFORE, BONUS chapter about deploying to heroku.

And the best part:

IT IS FREE TO DOWNLOAD!!!

Download it now

About

I'm Miguel Cobá. I write about Elixir, Elm, Software Development, and eBook writing.

• Follow me on Twitter
• Subscribe to my newsletter
• Read all my articles on my blog
• Get my books:
  • 100 Elixir Tips [FREE]
  • Deploying Elixir [FREE]
  • Deploying Elixir: Advanced Topics
  • eBook Writing Workflow for Developers

I'll use the Elixir Release we did before.

Configure the app for Gigalixir

Edit the runtime.exs file and add the url: line to the Endpoint section:

  config :saturn, SaturnWeb.Endpoint,
    url: [host: System.get_env("APP_NAME") <> ".gigalixirapp.com", port: 443],
    http: [
      # Enable IPv6 and bind on all interfaces.
      # Set it to  {0, 0, 0, 0, 0, 0, 0, 1} for local network only access.
      # See the documentation on https://hexdocs.pm/plug_cowboy/Plug.Cowboy.html
      # for details about using IPv6 vs IPv4 and loopback vs public addresses.
      ip: {0, 0, 0, 0, 0, 0, 0, 0},
      port: String.to_integer(System.get_env("PORT") || "4000")
    ],
    secret_key_base: secret_key_base

Add the config for the buildpacks that Gigalixir uses to compile and deploy the application.

Create a file named elixir_buildpack.config on the root of the project and put this:

erlang_version=24.1.2
elixir_version=1.12.3

Create a phoenix_static_buildpack.config file in the root and put this:

node_version=14.15.4

Create a file in assets/package.json and put this:

{
    "scripts": {
        "deploy": "cd .. && mix assets.deploy && rm -f _build/esbuild"
    }
}

This one is to use npm to use mix to use esbuild to compile our assets and after deploying them, delete the intermediate files!

Create a .buildpacks file and put this:

https://github.com/HashNuke/heroku-buildpack-elixir
https://github.com/gjaldon/heroku-buildpack-phoenix-static
https://github.com/gigalixir/gigalixir-buildpack-releases.git

Ok, enough changes. I am creating a branch named gigalixir-deployment and committing all these changes to it:

git checkout -b gigalixir-deployment
git add .
git commit -m "Deploying to Gigalixir"
git push -u origin gigalixir-deployment

Gigalixir setup

Install the gigalixir CLI

brew tap gigalixir/brew
brew install gigalixir

Crete an account in Gigalixir and login

gigalixir signup
gigalixir login

Create a Gigalixir application:

export APP_NAME=$(gigalixir create)

Created app: harsh-some-cats.
Set git remote: gigalixir.

Verify your app is created:

gigalixir apps

[
  {
    "cloud": "gcp",
    "region": "v2018-us-central1",
    "replicas": 0,
    "size": 0.3,
    "stack": "gigalixir-20",
    "unique_name": "harsh-some-cats",
    "version": 2
  }
]

This command also creates a new remote in your git repo, pointing to gigalixir. Check it:

git remote -v

gigalixir    https://git.gigalixir.com/harsh-some-cats.git/ (fetch)
gigalixir    https://git.gigalixir.com/harsh-some-cats.git/ (push)

Provision a database

Let's create a database for our app. This is the free plan so is not suitable for production but it will be good for testing.

gigalixir pg:create --free

You can check if the database is running with this command:

gigalixir pg

Finally, let's see the environment variables that Gigalixir created for us:

gigalixir config

You should see something like this:

{
  "DATABASE_URL": "ecto://<some user>:<some password>@postgres-free-tier-v2020.gigalixir.com:5432/<some database name>",
  "POOL_SIZE": "2"
}

As you see DATABASE_URL and POOL_SIZE are automatically created. Our app also needs the SECRET_KEY_BASE, PORT and APP_NAME but those will be automatically provided when the app starts, we don't need to worry about them.

Deploy the application

To deploy to Gigalixir, we must push our master branch to the gigalixir remote. Gigalixir only builds the master branch and ignore others. We use gigalixir-deployment as our main branch, so we need to push our branch changes to the master branch on the gigalixir remote:

git push gigalixir gigalixir-deployment:master

This will start the building of the release on the gigalixir build servers.

You can see the status with gigalixir ps:

{
  "cloud": "gcp",
  "pods": [
    {
      "lastState": {},
      "name": "harsh-some-cats-77d577b7cd-2s9c2",
      "sha": "5dd02b3af6fc7958615b9ee2cbcd6af934845506",
      "status": "Healthy",
      "version": "4"
    }
  ],
  "region": "v2018-us-central1",
  "replicas_desired": 1,
  "replicas_running": 1,
  "size": 0.3,
  "stack": "gigalixir-20",
  "unique_name": "harsh-some-cats"
}

When the build is finished, the app is deployed and started and you'll see log lines like these:

remote: Creating release.
remote: Starting zero-downtime rolling deploy.
remote: Please wait a minute for the new instance(s) to roll out and pass health checks.
remote: For troubleshooting, See:      http://gigalixir.readthedocs.io/en/latest/main.html#troubleshooting
remote: For help, contact:             help@gigalixir.com
remote: Try hitting your app with:     curl https://harsh-some-cats.gigalixirapp.com/
remote: Check your app logs with:      gigalixir logs -a harsh-some-cats
remote: Check deploy status with:      gigalixir ps -a harsh-some-cats
remote: Updated property [core/account].
To https://git.gigalixir.com/harsh-some-cats.git/
 * [new branch]      gigalixir-deployment -> master

You can open your app in your browser by running:

gigalixir open

And you'll see the familiar home screen:

Opening a remote console and running migrations

First we need to add our ssh public key so that we can connect securely:

gigalixir account:ssh_keys:add "$(cat ~/.ssh/id_rsa.pub)"

Opening a remote console

gigalixir ps:remote_console

Erlang/OTP 24 [erts-12.1.2] [source] [64-bit] [smp:8:1] [ds:8:1:10] [async-threads:1] [jit]

Interactive Elixir (1.12.3) - press Ctrl+C to exit (type h() ENTER for help)
iex(harsh-some-cats@10.56.18.206)1> ls
.bashrc           .profile          .profile.d        bin               erts-12.1.2
lib               releases          saturn.tar.gz

Running migrations

First, let's watch the logs:

gigalixir logs

Then, in another terminal, run:

gigalixir ps:migrate

Connection to v2018-us-central1.gcp.ssh.gigalixir.com closed.

You should see something like this:

2021-11-02T22:45:07.937626+00:00 harsh-some-cats[b'harsh-some-cats-77d577b7cd-2s9c2']: web.1  | 22:45:07.937 [info] Running SaturnWeb.Endpoint with cowboy 2.9.0 at :::4000 (http)
2021-11-02T22:45:07.938503+00:00 harsh-some-cats[b'harsh-some-cats-77d577b7cd-2s9c2']: web.1  | 22:45:07.938 [info] Access SaturnWeb.Endpoint at http://harsh-some-cats.gigalixirapp.com:443
2021-11-02T22:47:27.795085+00:00 harsh-some-cats[b'harsh-some-cats-77d577b7cd-2s9c2']: web.1  | 22:47:27.794 request_id=6ae142c1367400361510a08fb8000235 [info] GET /
2021-11-02T22:47:27.799510+00:00 harsh-some-cats[b'harsh-some-cats-77d577b7cd-2s9c2']: web.1  | 22:47:27.798 request_id=6ae142c1367400361510a08fb8000235 [info] Sent 200 in 4ms
2021-11-02T22:48:38.509918+00:00 harsh-some-cats[b'harsh-some-cats-77d577b7cd-2s9c2']: web.1  | 22:48:38.508 request_id=bbc52891b40f8bb8ec1c6e96b766c8fd [info] GET /
2021-11-02T22:48:38.509940+00:00 harsh-some-cats[b'harsh-some-cats-77d577b7cd-2s9c2']: web.1  | 22:48:38.509 request_id=bbc52891b40f8bb8ec1c6e96b766c8fd [info] Sent 200 in 1ms
2021-11-02T22:53:33.926744+00:00 harsh-some-cats[b'harsh-some-cats-77d577b7cd-2s9c2']: web.1  | 22:53:33.926 [info] Migrations already up

That's it.

Source code

The source code for the saturn project is open source under the MIT license. Use the gigalixir-deployment branch.

About

I'm Miguel Cobá. I write about Elixir, Elm, Software Development, and eBook writing.

• Follow me on Twitter
• Subscribe to my newsletter
• Read all my articles on my blog
• Get my books:
  • 100 Elixir Tips [FREE]
  • Deploying Elixir [FREE]
  • Deploying Elixir: Advanced Topics
  • eBook Writing Workflow for Developers

Photo by Christina @ wocintechchat.com on Unsplash

Prepare Elixir Release for deploying to render.com

We only need to do a single change to our runtime.exs file. Add the url: line to the

  config :saturn, SaturnWeb.Endpoint,
    url: [host: System.get_env("RENDER_EXTERNAL_HOSTNAME") || "localhost", port: 80],
    http: [
      # Enable IPv6 and bind on all interfaces.
      # Set it to  {0, 0, 0, 0, 0, 0, 0, 1} for local network only access.
      # See the documentation on https://hexdocs.pm/plug_cowboy/Plug.Cowboy.html
      # for details about using IPv6 vs IPv4 and loopback vs public addresses.
      ip: {0, 0, 0, 0, 0, 0, 0, 0},
      port: String.to_integer(System.get_env("PORT") || "4000")
    ],
    secret_key_base: secret_key_base

I am creating a branch named render-deployment and committing all these changes to it:

git checkout -b render-deployment
git add .
git commit -m "Config for render.com"
git push -u origin render-deployment

Create and configure your render.com account

Create an account in render.com and log in. We are going to create two services, one for the database and one for the Elixir Release. We must be sure that both services are created in the same region so that they share the same private network and can communicate and establish connections using their assigned network names. If we fail to do this, you'll see errors in the logs saying that the database domain name is non-existant (:nxdomain error).

For this article, I'm going to put both in the Oregon, USA region. Be sure to use the same one.

Create the Database

Create a database service by clicking the New + button on the dashboard:

Select one name for this service (I'm using Saturn DB) and ensure the Oregon, USA region is selected. Leave the other fields to their defaults and select the Free plan. Then click the Create Database to provision this service.

In the next page you'll see the details of the database. We need to copy one value from here, the connection string that we'll use for the DATABASE_URL environment variable when we create the Elixir Release service.

Click the copy button on the Internal Connection String

Create the Web Service

Let's create a Web Service with the New + button:

You'll need to connect your GitHub or GitLab account:

and select the repository you want to deploy:

Let's configure it. Set a name for your service (I'm using Saturn here). You'll see that the Dockerfile was detected and the Environment is preselected to Docker. Ensure you're using the Oregon, USA region, otherwise both services won't be able to connect each other. Select the branch you want to deploy (in my case is render-deployment) and select the Free plan.

We need to add a couple of environment variables.

Create a DATABASE_URL and paste the value you copied from the Internal Connection String field in the Database service config page.

Then add another named SECRET_KEY_BASE. Run this command on the terminal and use the result as the value for it:

mix phx.gen.secret

You can now create the Web Service. You'll see the status of the provisioning and when it is ready you'll see a green Live button and a line on the logs saying that the Endpoint is listening for requests:

If you click on the URL generated for your Web Service you'll see your Elixir Release app running on Render.com infrastructure:

Running Migrations and connecting to the running instance

This is something I couldn't do so far. The UI render provides has no way to connect to the running container. One way to run the migrations would be:

1. Build the Elixir Release locally as shown here
2. Copy the External Connection String from the database configuration page
3. Set a DATABASE_URL environment variable to that value
4. Run the _build/prod/rel/saturn/bin/saturn eval "Saturn.Release.migrate" command

Source code

The source code for the saturn project is open source under the MIT license. Use the render-deployment branch.

About

I'm Miguel Cobá. I write about Elixir, Elm, Software Development, and eBook writing.

• Follow me on Twitter
• Subscribe to my newsletter
• Read all my articles on my blog
• Get my books:
  • 100 Elixir Tips [FREE]
  • Deploying Elixir [FREE]
  • Deploying Elixir: Advanced Topics
  • eBook Writing Workflow for Developers

Photo by Mueen Agherdien on Unsplash

Prepare Elixir Release for deploying to Fly.io

Fly.io uses IPv6 in all their internal networks. So we need to configure our app to use IPv6 if we want to connect the app to the database.

Run this command to generate, among others, the rel/env.sh.eex file:

mix release.init

This file runs just before starting our application. It configures environment variables dynamically. Set the contents of the file to this:

#!/bin/sh

ip=$(grep fly-local-6pn /etc/hosts | cut -f 1)
export RELEASE_DISTRIBUTION=name
export RELEASE_NODE=$FLY_APP_NAME@$ip
export ELIXIR_ERL_OPTIONS="-proto_dist inet6_tcp"

This file gets the IPv6 assigned by fly.io on startup and assigns it to a variable. Then it uses that variable, along with the FLY_APP_NAME environment variable that fly.io automatically provides, to set another environment variable RELEASE_NODE. This will be used as a unique name for the node that our app is running in. The last line configures the BEAM virtual machine to use IPv6.

Let's modify the config/runtime.exs file.

Change the Saturn.Repo config to:

config :saturn, Saturn.Repo,
    # ssl: true,
    socket_options: [:inet6],
    url: database_url,
    pool_size: String.to_integer(System.get_env("POOL_SIZE") || "10")

Change the SaturnWeb.Endpoint to

  app_name =
    System.get_env("FLY_APP_NAME") ||
      raise "FLY_APP_NAME not available"

  config :saturn, SaturnWeb.Endpoint,
    url: [host: "#{app_name}.fly.dev", port: 80],
    http: [
      # Enable IPv6 and bind on all interfaces.
      # Set it to  {0, 0, 0, 0, 0, 0, 0, 1} for local network only access.
      # See the documentation on https://hexdocs.pm/plug_cowboy/Plug.Cowboy.html
      # for details about using IPv6 vs IPv4 and loopback vs public addresses.
      ip: {0, 0, 0, 0, 0, 0, 0, 0},
      port: String.to_integer(System.get_env("PORT") || "4000")
    ],
    secret_key_base: secret_key_base

Add a .dockerignore file to the root of the project:

assets/node_modules/
deps/

Modify the Dockerfile and change the line that copies the runtime.exs file to this:

# copy runtime configuration file
COPY rel rel
COPY config/runtime.exs config/

I am creating a branch named fly-io-deployment and committing all these changes to it:

git checkout -b fly-io-deployment
git add .
git commit -m "Deploying to fly.io"
git push -u origin fly-io-deployment

Create and configure your Fly.io account

Install flyctl

brew install superfly/tap/flyctl

Sign up to fly.io

If you don't have a fly.io account, create one

flyctl auth signup

Login to fly.io

If you already have a fly.io account, login

flyctl auth login

Create a Fly.io app

Before launching the app ensure you have added a credit card to your organization by visiting https://fly.io/organizations/personal and adding one. Otherwise, the next command won't work.

Once you're ready, run this command:

fly launch

It will ask you some things to configure your app in fly.io. Leave the App name blank in order to get a random name for it. Pick a region close to where you live and make sure that you answer no to the question about deploying now.

You should see something similar to this:

fly launch
Creating app in /Users/mcoba/Code/saturn
Scanning source code
Detected a Dockerfile app
? App Name (leave blank to use an auto-generated name):
Automatically selected personal organization: Miguel Cobá
? Select region: mad (Madrid, Spain)
Created app damp-paper-3277 in organization personal
Wrote config file fly.toml
? Would you like to deploy now? No
Your app is ready. Deploy with `flyctl deploy`

Open the fly.toml file that flyctl created in the root of the project. Change the kill_signal to:

kill_signal = "SIGTERM"

and add a [deploy] section after [env]

[env]

[deploy]
  release_command = "eval Saturn.Release.migrate"

change the internal_port to:

  internal_port = 4000

Set secrets on Fly.io

We need to create some secrets in Fly.io infrastructure to be used when the app starts.

fly secrets set SECRET_KEY_BASE=$(mix phx.gen.secret)

Create database

Create a database for the app. Aswer the questions leaving the app name blank to get a random name and ensure you select the smallest VM size.

fly postgres create

You should see something similar to this:

fly postgres create

? App Name:
Automatically selected personal organization: Miguel Cobá
? Select region: mad (Madrid, Spain)
? Select VM size: shared-cpu-1x - 256
? Volume size (GB): 10
Creating postgres cluster  in organization personal
Postgres cluster still-sun-6781 created
  Username:    postgres
  Password:   <some big password>
  Hostname:    still-sun-6781.internal
  Proxy Port:  5432
  PG Port: 5433
Save your credentials in a secure place, you won't be able to see them again!

Monitoring Deployment

2 desired, 2 placed, 0 healthy, 0 unhealthy [health checks: 6 total, 1 passing,
2 desired, 2 placed, 0 healthy, 0 unhealthy [health checks: 6 total, 1 passing,
2 desired, 2 placed, 0 healthy, 0 unhealthy [health checks: 6 total, 1 passing,
2 desired, 2 placed, 0 healthy, 0 unhealthy [health checks: 6 total, 2 passing,
2 desired, 2 placed, 0 healthy, 0 unhealthy [health checks: 6 total, 3 passing,
2 desired, 2 placed, 0 healthy, 0 unhealthy [health checks: 6 total, 4 passing,
2 desired, 2 placed, 0 healthy, 0 unhealthy [health checks: 6 total, 5 passing,
2 desired, 2 placed, 2 healthy, 0 unhealthy [health checks: 6 total, 6 passing]
--> v0 deployed successfully

Connect to postgres
Any app within the personal organization can connect to postgres using the above credentials and the hostname "still-sun-6781.internal."
For example: postgres://postgres:<the big password>@still-sun-6781.internal:5432

See the postgres docs for more information on next steps, managing postgres, connecting from outside fly:  https://fly.io/docs/reference/postgres/

Take note of the generated database name, you'll need it in the next step. Mine is: still-sun-6781.

What remains is to connect the Elixir Release app to the PostgreSQL app. Run this command but use your own database name. This will create a new postgres user and password to connect from the Elixir Release to the PostgreSQL database:

fly postgres attach --postgres-app still-sun-6781

You'll see something like this:

fly postgres attach --postgres-app still-sun-6781

Postgres cluster still-sun-6781 is now attached to damp-paper-3277
The following secret was added to damp-paper-3277:
  DATABASE_URL=postgres://<some new user>:<some new password>@still-sun-6781.internal:5432/damp_paper_3277?sslmode=disable

As you can see, this automatically created a secret with the DATABASE_URL that we were missing.

Deploy to Fly.io

Do the deployment:

fly deploy

This will start the Docker image building, push it to fly.io's registry and then will deploy a container based on that image and will provide the secrets we configure it to start it. After lots of output logs you should see something like this:

==> Release command
Command: eval Saturn.Release.migrate
     Starting instance
     Configuring virtual machine
     Pulling container image
     Unpacking image
     Preparing kernel init
     Configuring firecracker
     Starting virtual machine
     Starting init (commit: 50ffe20)...
     Preparing to run: `bin/saturn eval Saturn.Release.migrate` as elixir
     2021/10/29 23:19:47 listening on [fdaa:0:37f6:a7b:2656:f312:7c7b:2]:22 (DNS: [fdaa::3]:53)
     Reaped child process with pid: 561 and signal: SIGUSR1, core dumped? false
     23:19:50.604 [info] Migrations already up
     Main child exited normally with code: 0
     Reaped child process with pid: 563 and signal: SIGUSR1, core dumped? false
     Starting clean up.
Monitoring Deployment

1 desired, 1 placed, 1 healthy, 0 unhealthy [health checks: 1 total, 1 passing]
--> v1 deployed successfully

As you see the deployment was executed correctly and it ran the migrations. Now let's visit the app.

fly open

A browser is opened and you should be presented with your app, running on Fly.io infrastructure:

Bonus

Connect to the running node with IEx

We need to configure a secure ssh tunnel to the container running in fly.io.

fly ssh establish
fly ssh issue

Answer with your email and select a place to save your private keys. If you already use ssh for other connections you can save it to the same $HOME/.ssh/ directory. I got this:

fly ssh establish
Automatically selected personal organization: Miguel Cobá
Establishing SSH CA cert for organization personal
New organization root certificate:
ssh-ed25519-cert-v01@openssh.com <some big value>

fly ssh issue
? Email address for user to issue cert:  miguel.coba@gmail.com

!!!! WARNING: We're now prompting you to save an SSH private key and certificate       !!!!
!!!! (the private key in "id_whatever" and the certificate in "id_whatever-cert.pub"). !!!!
!!!! These SSH credentials are time-limited and handling them in files is clunky;      !!!!
!!!! consider running an SSH agent and running this command with --agent. Things       !!!!
!!!! should just sort of work like magic if you do.                                    !!!!
? Path to store private key:  ~/.ssh/id_fly_io
? Path to store private key:  /Users/mcoba/.ssh/.id_fly_io
Wrote 24-hour SSH credential to /Users/mcoba/.ssh/.id_fly_io, /Users/mcoba/.ssh/.id_fly_io-cert.pub

You can now connect to the container with fly ssh console and connect to the erlang node with app/bin/saturn remote:

fly ssh console
Connecting to damp-paper-3277.internal... complete
/ # cd ~
/home/elixir # ls
app
/home/elixir # app/bin/saturn remote
Erlang/OTP 24 [erts-12.1.2] [source] [64-bit] [smp:1:1] [ds:1:1:10] [async-threads:1] [jit:no-native-stack]

Interactive Elixir (1.12.3) - press Ctrl+C to exit (type h() ENTER for help)

That's it.

Source code

The source code for the saturn project is open source under the MIT license. Use the fly-io-deployment branch.

About

I'm Miguel Cobá. I write about Elixir, Elm, Software Development, and eBook writing.

• Follow me on Twitter
• Subscribe to my newsletter
• Read all my articles on my blog
• Get my books:
  • 100 Elixir Tips [FREE]
  • Deploying Elixir [FREE]
  • Deploying Elixir: Advanced Topics
  • eBook Writing Workflow for Developers

Photo by Jan Ranft on Unsplash

There are several approaches to this:

• Let DigitalOcean access directly your GitHub or GitLab repository
• Put the image in Docker public registry where DigitalOcean can fetch it
• Put the image in DigitalOcean own Container Registry

In this article we'll use the first approach

Create a repository

I'll show how to do it in GitHub but should be similarly easy in GitLab

GitHub

Create a new repository

push the code:

git remote add origin git@github.com:miguelcoba/saturn.git
git branch -M main
git push -u origin main

Prepare to deploy to DigitalOcean

DigitalOcean requires some changes to our code in order to correctly connect to the DB from our Docker application.

So far we have been establishing a plain, unencrypted connections between the application and the database. This won't work for DigitalOcean as they strictly enforce SSL when connecting to the database.

In our case, that means that we need to enable SSL on the application side every time we open a connection to the database.

First thing is to configure the Repo to use ssl. Open runtime.ex and uncomment the ssl: true line on the Repo section:

  config :saturn, Saturn.Repo,
    ssl: true,
    # socket_options: [:inet6],
    url: database_url,
    pool_size: String.to_integer(System.get_env("POOL_SIZE") || "10")

This will enable SSL for all the connections in the pool that the repository uses to connect to the database.

We need to change the Saturn.Release module that we use to run the migrations, because that also opens a connection to the database and we need it to start the ssl application at the beginning.

Change the load_app function in the Saturn.Release module to be like this:

  defp load_app do
    Application.ensure_all_started(:ssl)
    Application.load(@app)
  end

This will ensure the ssl application is started before trying to connect to the database to run the migrations.

We need to commit this changes to the repository. DigitalOcean works by deploying a specific branch from our repository and building the Docker image with what it finds in that branch. If you plan to use Docker as your production environment then is fine to commit this to the main branch and setup DigitalOcean to use the main branch for deployment. For this example, I will use a different branch to commit these changes and configure DigitalOcean to use that branch. I'll use a branch named digital-ocean-deployment

git checkout -b digital-ocean-deployment
git add .
git commit -m "Setup for deploying to DigitalOcean"
git push -u origin digital-ocean-deployment

Creating an App in DigitalOcean

Let's do the deployment to DigitalOcean.

Go to your DigitalOcean dashboard and create a new App:

Select GitHub on the "Choose Source" screen:

The first time you do this, GitHub will require you to authorize DigitalOcean to access your repositories. Follow the instructions and give access to the repository you just created

You'll get back to the "Choose source" and you'll see the list of authorized repositories there

Select your repository and you'll see that the main branch is selected automatically

If you're using main that's fine. For this article, I'll change it to the digital-ocean-deployment branch:

Click Next to move to the "Configure your app" page:

We need to configure the environment variables that DigitalOcean will provide to our Docker container in runtime. We can omit the PORT environment variable as DigitalOcean automatically provides it. We have to create the POOL_SIZE, DATABASE_URL, and SECRET_KEY_BASE. Let's start with the last one.

Click on the Edit link in the "Environment Variables" section. Then, in a terminal execute this command:

mix phx.gen.secret

Add a variable named SECRET_KEY_BASE and put the value you got from the command in it. Check the "Encrypt" option.

Now create a POOL_SIZE with value 2 and a DATABASE_URL with value ${db.DATABASE_URL}. These two don't need to be encrypted.

We need to create a database. Click on the "Add a Database" button:

Accept the default values for the database and now you're ready to create the App in DigitalOcean.

One thing to note: the DATABASE_URL environment variable is referring to this database we just created. It will be replaced with the correct value to connect to this database when the application starts.

Click next to proceed to the "Name your web service" page:

Accept the defaults and click Next. You're now in the "Finalize and launch" page:

Click the "Launch Basic App" and you'll see your app being created.

DigitalOcean will now:

1. Access your repository
2. Clone it,
3. Detect the Dockerfile in it
4. Build a Docker image from it
5. Push it to their own Container Registry
6. Provision the database
7. Set up environment variables
8. Provision a server to run the Docker container
9. Run the application

You can check the progress by inspecting the logs:

At the end you'll have your app deployed:

Run the migrations

You can access the elixir container console and run the migrations. Go to the "Console" tab and there evaluate the following command:

bin/saturn eval "Saturn.Release.migrate"

You'll see something like this:

Finally, visit the application by clicking the "Live App" button:

You'll see your app running in DigitalOcean's infrastructure:

We are done!

Source code

The source code for the saturn project is open source under the MIT license. Use the digital-ocean-deployment branch.

About

I'm Miguel Cobá. I write about Elixir, Elm, Software Development, and eBook writing.

• Follow me on Twitter
• Subscribe to my newsletter
• Read all my articles on my blog
• Get my books:
  • 100 Elixir Tips [FREE]
  • Deploying Elixir [FREE]
  • Deploying Elixir: Advanced Topics
  • eBook Writing Workflow for Developers

Photo by Kris-Mikael Krister on Unsplash

Prerequisites

Follow the previous guides to prepare your Phoenix 1.6 image to use Elixir Releases.

Build process

Docker builds images in steps, each step generating an image layer as a result. If you organize your steps wisely, you can avoid rebuilding layers and the build process will be faster. For example, if one step downloads dependencies for the project and the next step compiles the code, the first time you build it, both steps will be executed and two layers will be generated. But if you later make a change in the source code and rebuild the Docker image, you'll see that the second step will be executed but the first one will be skipped. A new second layer will be generated too, to replace the old second one. The first step won't be executed, because no changes to the dependencies were made and therefore there is no need to regenerate its image layer. The net effect is that you get a faster build time the second time.

The Docker image we are going to create for the Phoenix app will take advantage of this.

There is an additional feature we'll use. Docker allows you group steps into build stages. One build stage can refer to previous build stages. We will use one stage to do build the elixir release and another stage to run the release.

Docker image stage for building the release

First let's create a Dockerfile in the root of the project:

ARG MIX_ENV="prod"

# build stage
FROM hexpm/elixir:1.12.3-erlang-24.1.2-alpine-3.14.2 AS build

# install build dependencies
RUN apk add --no-cache build-base git python3 curl

The first line declares an argument that can be passed to the docker builder at build time: MIX_ENV. Similar to command line arguments, it allows us to change the behavior of the build. We are also giving it a default value of "prod", so that we don't have to specify it every time.

Second line starts a build stage specifying a base image to build upon. It names the build stage as build. The base image we use has the same versions we use in our project: Elixir 1.12.3 and Erlang 24.1.2. Alpine is a small linux distribution with fast startup time and small memory usage.

The RUN command installs some basic build tools into the Alpine Linux image, creating a docker image layer as a result.

# sets work dir
WORKDIR /app

# install hex + rebar
RUN mix local.hex --force && \
    mix local.rebar --force

This creates a working directory for all the subsequent commands to work on. Then installs hex and rebar.
As we seldom change the build tools or the hex or rebar install commands, this image layer won't be regenerated often.

ARG MIX_ENV
ENV MIX_ENV="${MIX_ENV}"

# install mix dependencies
COPY mix.exs mix.lock ./
RUN mix deps.get --only $MIX_ENV

This new layer will contain the mix dependencies for the environment we choose. You can see we are setting the MIX_ENV environment variable to the value of the MIX_ENV variable. Although they have the same name, they are different things. One is the value we pass from outside to the docker when we start the build. The other is an environment variable that will exist during the image build process.

But there is a catch. The MIX_ENV argument we declared in the first line of the Docker file doesn't exist anymore after the FROM line. We can use its default value if we redeclare it after a FROM as we are doing here. The effect is that the docker variable MIX_ENV will have a value again in this build stage.

Note that if we change the mix.ex or mix.lock files, this step will generate a new layer. Otherwise it will remain unchanged.

# copy compile configuration files
RUN mkdir config
COPY config/config.exs config/$MIX_ENV.exs config/

# compile dependencies
RUN mix deps.compile

New layer. This one copies the compile time configuration files and then start the dependencies compilation. Again, if we don't change the config files content, this layer won't be regenerated.

# copy assets
COPY priv priv
COPY assets assets

# Compile assets
RUN mix assets.deploy

This step compiles the web assets. Again, no changes in priv/ or assets/ means no new layer is created here.

Now we can compile the project binaries.

# compile project
COPY lib lib
RUN mix compile

We already have a compiled binary, but Elixir Release allows to configure things for runtime.

# copy runtime configuration file
COPY config/runtime.exs config/

# assemble release
RUN mix release

So this step is generating a new layer that will only be rebuilt if there are changes in the runtime configuration. The output of this step is an assembled release.

At this point we have built our Elixir/Phoenix release. But we only need the output binaries that were generated into the _build/prod/rel/saturn directory. Everything else is of no use for us in the production server. We can discard anything else for running the release.

Docker image stage for running the release

We will use a new image stage for running the release. The release, as the Elixir documentation says, has anything required to run the project, including the Erlang virtual machine. That means that the hosting image doesn't need to have it installed separately. That's an extra opportunity to shrink the image size.

# app stage
FROM alpine:3.14.2 AS app

ARG MIX_ENV

# install runtime dependencies
RUN apk add --no-cache libstdc++ openssl ncurses-libs

This starts a new build stage named app. It uses a plain Alpine Linux base image without Elixir in it. Next, it redeclares the MIX_ENV docker variable so that it can be used later.
Then installs the runtime dependencies and nothing else.

ENV USER="elixir"

WORKDIR "/home/${USER}/app"

For security reasons we will to run the release binary using a non-privileged user in the Linux Alpine image. We are going to create a user named elixir for that. We declare an environment variable USER and the work dir pointing to a directory inside that user home. This user doesn't exist in the alpine base image we are using. Let's create it:

# Create  unprivileged user to run the release
RUN \
  addgroup \
   -g 1000 \
   -S "${USER}" \
  && adduser \
   -s /bin/sh \
   -u 1000 \
   -G "${USER}" \
   -h "/home/${USER}" \
   -D "${USER}" \
  && su "${USER}"

With the user created we can copy the build stage binaries into this build stage:

# run as user
USER "${USER}"

# copy release executables
COPY --from=build --chown="${USER}":"${USER}" /app/_build/"${MIX_ENV}"/rel/saturn ./

As you can see, we are switching to the newly created user and copying the assembled release into the workdir and changing ownership of the files to the new user.

ENTRYPOINT ["bin/saturn"]

CMD ["start"]

The last thing is to tell docker to run the container as an executable by telling it which command to run in ENTRYPOINT. The CMD specify default parameters to pass to the command in ENTRYPOINT in case we don't set any when running the container.

The Dockerfile should be like this:

ARG MIX_ENV="prod"

# build stage
FROM hexpm/elixir:1.12.3-erlang-24.1.2-alpine-3.14.2 AS build

# install build dependencies
RUN apk add --no-cache build-base git python3 curl

# sets work dir
WORKDIR /app

# install hex + rebar
RUN mix local.hex --force && \
    mix local.rebar --force

ARG MIX_ENV
ENV MIX_ENV="${MIX_ENV}"

# install mix dependencies
COPY mix.exs mix.lock ./
RUN mix deps.get --only $MIX_ENV

# copy compile configuration files
RUN mkdir config
COPY config/config.exs config/$MIX_ENV.exs config/

# compile dependencies
RUN mix deps.compile

# copy assets
COPY priv priv
COPY assets assets

# Compile assets
RUN mix assets.deploy

# compile project
COPY lib lib
RUN mix compile

# copy runtime configuration file
COPY config/runtime.exs config/

# assemble release
RUN mix release

# app stage
FROM alpine:3.14.2 AS app

ARG MIX_ENV

# install runtime dependencies
RUN apk add --no-cache libstdc++ openssl ncurses-libs

ENV USER="elixir"

WORKDIR "/home/${USER}/app"

# Create  unprivileged user to run the release
RUN \
    addgroup \
    -g 1000 \
    -S "${USER}" \
    && adduser \
    -s /bin/sh \
    -u 1000 \
    -G "${USER}" \
    -h "/home/${USER}" \
    -D "${USER}" \
    && su "${USER}"

# run as user
USER "${USER}"

# copy release executables
COPY --from=build --chown="${USER}":"${USER}" /app/_build/"${MIX_ENV}"/rel/saturn ./

ENTRYPOINT ["bin/saturn"]

CMD ["start"]

Now that we have a Dockerfile ready, let's build the image:

In the root of the project run this:

docker image build -t elixir/saturn .

This will build the image following the Dockerfile commands and will tag it with the name elixir/saturn. You'll see a lot of messages while it is being built. Also, the first time you run it will take some time as it will download the alpine images we specified in the FROM commands but the following times it will use a local cache and it will be faster.

You can check your local images with:

docker image list
REPOSITORY                      TAG            IMAGE ID       CREATED         SIZE
elixir/saturn                   latest         02d3d6d963f5   5 seconds ago   23.3MB

Now it is time to run a container based on this image we created. So far we have been using the PostgreSQL database installed locally, and we used a DATABASE_URL like this:

export DATABASE_URL=ecto://postgres:postgres@localhost/saturn_dev

But that is not going to work when we start the container for our app. The localhost address is going to point to the same linux instance the app is running in and that container has no PostgreSQL installed. We need to change the url to point to a database that is outside the docker image.

One option is to use the IP address our laptop has because that is accessible from the docker container. You could do something like this:

ifconfig en0
en0: flags=8863<UP,BROADCAST,SMART,RUNNING,SIMPLEX,MULTICAST> mtu 1500
    options=400<CHANNEL_IO>
    ether 6c:96:cf:dd:fd:3b
    inet6 fe80::c61:da63:14a3:73af%en0 prefixlen 64 secured scopeid 0x4
    inet 192.168.1.135 netmask 0xffffff00 broadcast 192.168.1.255
    nd6 options=201<PERFORMNUD,DAD>
    media: autoselect
    status: active

and use that IP address in the DATABASE_URL. But we'll also need to configure PostgreSQL pg_hba.conf to accept connection from hosts other than localhost or relax the security options, because the default install with brew in macOS is restricted to connections coming from localhost and the security is set to trust.

I am going to do it in other way: using docker containers. I'll start a docker container for PostgreSQL and a docker container for our Elixir/Phoenix app. And I'll connect them using a virtual network. Then I'll run the migrations from the elixir app container.

Let's create a the virtual network:

docker network create saturn-network

And let's get a postgres docker image and boot it up binding it up to the virtual network I just created:

docker run -d --network saturn-network --network-alias postgres-server -e POSTGRES_PASSWORD=supersecret postgres

This command downloads the latest postgres docker image, binds it to the network we created, sets a DNS alias for it, postgres-server, sets an environment variable for the default user and, as we didn't say otherwise, will use the default username postgres. It also runs in detached mode.

Check that is running:

docker ps
CONTAINER ID   IMAGE      COMMAND                  CREATED          STATUS          PORTS      NAMES
8281f722c845   postgres   "docker-entrypoint.s…"   45 seconds ago   Up 44 seconds   5432/tcp   kind_mahavira

Let's connect to it to create the database. You need to use the CONTAINER ID you got from the docker ps (change the id to your own container id)

docker exec -it 8281f722c845 psql -U postgres

Create the production database, and exit the container:

docker exec -it 8281f722c845 psql -U postgres
psql (14.0 (Debian 14.0-1.pgdg110+1))
Type "help" for help.

postgres=# CREATE DATABASE saturn_prod;
CREATE DATABASE
postgres=# \q

Now lets set our environment variables to point to this docker postgres database.:

export DATABASE_URL=ecto://postgres:supersecret@postgres-server/saturn_prod

The other environment variables don't require change, but they need to be exported in the shell you're going to use to run the elixir app container. If you have opened a new terminal ensure you have export them all there.

Now we can start the elixir app:

docker container run -dp $PORT:$PORT -e POOL_SIZE -e PORT -e DATABASE_URL -e SECRET_KEY_BASE --network saturn-network  --name saturn elixir/saturn

You can see that we are mapping the port used inside the docker image to start the elixir release to the same port number on the running container. We pass the other environment variables as they are currently exported in the terminal. Notice that this docker container will bind to the saturn-network too, otherwise won't be able to reach the postgres-server host.

Finally, let's run the migrations. We will send the command to the elixir app container and that will run the migrations over the virtual network in the postgres container.

docker exec -it saturn bin/saturn eval "Saturn.Release.migrate"

You should see something like this:

12:14:32.135 [info] Migrations already up

You can now point to http://localhost:4001 and you'll be accessing your app, deployed as an Elixir Release, inside a docker container that is connected through a virtual network to another container with postgres running in it.

How cool is that?

Source code

The source code for the saturn project is open source under the MIT license. Use the main branch.

About

I'm Miguel Cobá. I write about Elixir, Elm, Software Development, and eBook writing.

• Follow me on Twitter
• Subscribe to my newsletter
• Read all my articles on my blog
• Get my books:
  • 100 Elixir Tips [FREE]
  • Deploying Elixir [FREE]
  • Deploying Elixir: Advanced Topics
  • eBook Writing Workflow for Developers

Photo by Ian Taylor on Unsplash

• 20211009 added "Migrations support" section

I'm going to prepare a Phoenix application for deployment.

Prerequisites

Ensure to have your Phoenix 1.6 app running locally.

Runtime configuration

When deploying to production is better to inject runtime info to the application when it starts instead of having that info hardcoded in the source code. We pass info to the app to affect the way it works depending on the environment we are deploying the app to (e.g. staging, production).

Elixir 1.11 has introduced a way to inject this runtime info easily with the config/runtime.exs config file. If you open that file you'll see that it obtains some values from environment variables. The default environment variables to configure are POOL_SIZE, PORT, DATABASE_URL and SECRET_KEY_BASE. We need to specify a value for those envvars if we want our deployment to work correctly.

For now we are going to test it locally, in our laptop. In a deployment service, like Gigalixir or Fly.io, those envvars are going to be provided when the app starts. We are going to do that manually here:

export POOL_SIZE=2
export PORT=4001
export DATABASE_URL=ecto://postgres:postgres@localhost/saturn_dev
export SECRET_KEY_BASE=$(mix phx.gen.secret)

I have my database named locally saturn_dev and the user and password the ones shown. You can see your own connection parameters in config/dev.exs

Build in production mode

Compile elixir code

We can now get the production dependencies:

mix deps.get --only prod
MIX_ENV=prod mix compile

Compile assets

If the project has JS, CSS or other assets you can also compile them with the esbuild wrapper that phoenix now uses:

MIX_ENV=prod mix assets.deploy

Test that the project starts in prod mode

By now, if you haven't closed the terminal, you'll have the previous envvars still defined. If you have closed the terminal, you need to set them again.

MIX_ENV=prod mix phx.server

If you go to http://localhost:4001/ you'll see the homepage of the app, but this time it is using the configuration that the config/runtime.exs read from the terminal when it started instead of using the config/dev.exs configuration. One thing you'll notice is that the LiveDashboard link is gone. This works only in dev mode.

Generate a release

We need to do an extra step before building the release using Elixir Releases. Open config/runtime.exs and uncomment the following line, in the section titled "Using releases"

config :saturn, SaturnWeb.Endpoint, server: true

This direct the app to start the webserver when running the release executable. When we used mix phx.server this was done for us. Now we need to explicitly enable it.

After saving those changes we can now generate the release:

MIX_ENV=prod mix release

Run the release

We can now run the release executable generated by the mix release task:

_build/prod/rel/saturn/bin/saturn start

If you go again to http://localhost:4001/ you'll see the app running, but this time from the self-contained bundle that the Elixir Releases generated for us.

Migrations support

There is one more thing before finishing. Right now we are using a database that was created by a mix ecto.create command. But the release we just generated has no support for running mix in production. There is no mix command anywhere inside the _build/prod/rel folder. So how are we going to create the database and to run the phoenix migrations? Good question. We need a workaround that is embedded in the application itself.

Create a file in lib/saturn/release.ex and put this content there:

defmodule Saturn.Release do
  @app :saturn

  def migrate do
    load_app()

    for repo <- repos() do
      {:ok, _, _} = Ecto.Migrator.with_repo(repo, &Ecto.Migrator.run(&1, :up, all: true))
    end
  end

  def rollback(repo, version) do
    load_app()
    {:ok, _, _} = Ecto.Migrator.with_repo(repo, &Ecto.Migrator.run(&1, :down, to: version))
  end

  defp repos do
    Application.fetch_env!(@app, :ecto_repos)
  end

  defp load_app do
    Application.load(@app)
  end
end

Save the file and regenerate the release:

MIX_ENV=prod mix release

Let's create a production database in our local postgres to simulate the production database in our production environment. Login to PostgreSQL locally create the database:

psql -U postgres -h localhost 
psql (13.4)
Type "help" for help.

postgres=# CREATE DATABASE saturn_prod;
CREATE DATABASE

You need to change the DATABASE_URL to point to the new database:

export DATABASE_URL=ecto://postgres:postgres@localhost/saturn_prod

And now you can run the migrations:

_build/prod/rel/saturn/bin/saturn eval "Saturn.Release.migrate"

and you should see something like this:

23:41:17.647 [info] Migrations already up

Now you can start the app and it will point to the saturn_prod database we just created:

_build/prod/rel/saturn/bin/saturn start

Go again to http://localhost:4001 and the app will work normally, but now the database is fully migrated.

That's it.

You can put the contents of the _build/prod/rel/saturn folder in your production server (as long as same architecture that the computer you used to assemble the release) and start it. You don't need anything else installed because this folder includes all the dependencies and binaries required to run the application. As long as you set the environment variables with correct values for production, this should work flawless.

You could do that manually, for example, by copying this folder to a DigitalOcean droplet or any other VPS provider, but there are better ways to do that.

I'll show you how to deploy to Gigalixir in a future post.

Cheers

About

I'm Miguel Cobá. I write about Elixir, Elm, Software Development, and eBook writing.

• Follow me on Twitter
• Subscribe to my newsletter
• Read all my articles on my blog
• Get my books:
  • 100 Elixir Tips [FREE]
  • Deploying Elixir [FREE]
  • Deploying Elixir: Advanced Topics
  • eBook Writing Workflow for Developers

Photo by Gautier Salles on Unsplash

Install asdf

Install asdf

Install plugins

asdf plugin-add erlang
asdf plugin-add elixir

Install dependencies

asdf install erlang 24.1.2
asdf global erlang 24.1.2
asdf install elixir 1.12.3-otp-24
asdf global elixir 1.12.3-otp-24

Install Phoenix

mix local.rebar --force
mix local.hex --force
mix archive.install hex phx_new 1.6.0 --force

Create the Phoenix application

mix phx.new saturn --install

Create DB and run application

cd saturn
mix ecto.create
mix phx.server

Visit your Phoenix LiveDashboard

Open http://localhost:4000/dashboard

Done.

About

I'm Miguel Cobá. I write about Elixir, Elm, Software Development, and eBook writing.

• Follow me on Twitter
• Subscribe to my newsletter
• Read all my articles on my blog
• Get my books:
  • 100 Elixir Tips [FREE]
  • Deploying Elixir [FREE]
  • Deploying Elixir: Advanced Topics
  • eBook Writing Workflow for Developers

Photo by Kyle Kempt on Unsplash