Write your own CRON with Elixir

Elixir is rising in popularity and I think this is mostly due to the amazing Phoenix Framework. But for once, let's forget about WEB development and write a simple software: a complete CRON.

If you are too lazy to read, you can directly check the code on Github

Features

We want to build something simple yet production-ready. Therefore we need to be able to:

define a list of tasks to run via a configuration file
have a task runner to execute the tasks
add the tasks to our Supervision Tree (so we can monitor them)
log the result of each task
important events should trigger emails
ship the CRON as a binary
have proper unit tests

In this article, we will focus on the 2 first items

Let's develop!

Setup the Elixir project

First things first. Let's create a new project:

$ mix new omycron --sup

This command creates an empty project. The --sup option also generates an application that is going to be the main hosts for our other processes - the first node of our supervision tree.

defmodule Omycron.Application do
  @moduledoc """
  Main supervisor for Omycron
  """
  use Application

  @impl true
  def start(_type, _args) do
    children = []

    opts = [strategy: :one_for_one, name: Omycron.Supervisor]
    Supervisor.start_link(children, opts)
  end
end

lib/omicron/application.ex

An Application is just a Supervisor that is automatically started by IEX by running iex -S mix. We will soon add more children.

The tasks configuration file

Now we need to define the format of our config file. I like the simplicity of TOML so this is what we are going to use.

We will define our list of tasks using a TOML array of table:

[[task]]
name = "Im alive"
interval = {second = 3}
command = "echo"
params = ["i'am alive"]

[[task]]
name = "list folder"
interval = {second = 10, minute = 1}
command = "ls"
params = ['-a', '-l']

[[task]]
name = "a new day"
interval = {hour = 24}
command = "date"

omycron.toml

For now, each task must have a name, a shell command, and an interval. Optionally, we can also send an array of parameters.

Now we created the tasks config file, we need a way to dynamically retrieve its pathname in our Elixir application. While we could use environment variables, Elixir provides a Config module that is much easier to maintain.

The config for an Elixir project is usually in config/config.exs. So let's create this file and add the following:

import Config

config :omycron,
  tasks_pathname: "./omycron.toml"

config/config.exs

You can now retrieve the pathname of the config file using this command:

Application.fetch_env!(:omycron, :tasks_pathname)

Setup the tasks storage

We want to parse the TOML once and then store the list of tasks. In Elixir, the right way to store such data is with an Agent. An agent is just a process that is able to hold a state and provide some helpers to set and retrieve its state - yup, you got it right, the right way to store data in Elixir is through a sub-process!

In a folder tasks, create a new file agent.ex and following the documentation, initialize an empty Agent:

defmodule Omycron.Tasks.Agent do
  @moduledoc """
  Store the tasks
  """
  use Agent

  def start_link([config_pathname]) do
    IO.puts("IM STARTING !")
    Agent.start_link(fn -> %{} end, name: __MODULE__)
  end
end

lib/omycron/tasks/agent.ex

As you can see, the first parameter of start_link is a function that should return the data to store. For now, it stores an empty map. All sub-processes started properly can also have a name (2nd parameter). It allows us to retrieve the process using its name instead of its PID.

If you tried to run iex -S mix, you might have noticed that our agent doesn't start. Since this is a sub-process, it needs to be added to our supervision tree. This is when our initial Application comes in handy! Edit omycron/application.ex:

# ...

def start(_type, _args) do
  children = [
    Omycron.Tasks.Agent.child_spec()
  ]
# ...
end

lib/omycron/application.ex

Now we added our newly created Agent as a sub-process of our main application. It will be automatically started by IEX. And since our Application supervisor is set with the strategy one_for_one, if anything happens to our Agent and it crashes, it will automatically be restarted!

Time to parse!

Now everything is ready to store our tasks, we need to write our parser. We will use the librarytoml_elixir. It's TOML 0.5 compliant. Not perfect but good enough.

Add toml_elixir as a dependency

  defp deps do
    [
      {:toml_elixir, "~> 2.0.0"}
    ]
  end

mix.exs

Once you have installed the dependency with mix deps.get, create a new file tasks/parser.exthat looks like this:

defmodule Omycron.Tasks.Parser do
   @moduledoc """
  Parse the TOML config file
  """

  defmodule __MODULE__.Task do
    @moduledoc """
    Define a struct that contains all the properties for a Task
    """
    defstruct name: nil, command: nil, params: [], interval: nil
  end

  @doc """
  Start the parsing
  """
  @spec parse(String.t()) :: {:ok, list(any)} | {:error, String.t()}
  def parse(pathname) do
    # Use toml_elixir to read the file
    TomlElixir.parse_file(pathname)
    # parse_content use the value returned by toml_elixir
    |> parse_content()
    # `tap` execute a function and returns the value passed as parameters
    # this is used temporary to print the result and still return the tasks
    |> tap(&IO.puts("Parsed tasks > #{inspect(&1)}"))
  end

  ## In case `toml_elixir` returns an error, we display a proper error message
  ## and exit the program
  defp parse_content({:error, error_type}) do
    case error_type do
      :enoent -> write_error("Config file is missing")
      _ -> write_error("Failed to parse config file")
    end

    write_error("Program exits.")
    exit(:shutdown)
  end

  ## If `toml_elixir` was successful, we parse the content
  ## we use pattern matching to direcly access the array of tasks
  defp parse_content({:ok, %{"task" => tasks}}) do
    parse_tasks(tasks, [])
  end

  ## parse_tasks loop trough all tasks (thanks to pattern matching)
  ## 	and process them one by one.
  ## Once its done, it returns `parsed_tasks`
  defp parse_tasks([], parsed_tasks), do: parsed_tasks

  defp parse_tasks([task | tail], parsed_tasks),
    do: parse_tasks(tail, [parse_task(task) | parsed_tasks])

  ## parse_task takes the raw data returned from `toml_elixir`
  ## and cast it into a Task struct defined earlier
  ## we use pattern matching to make sure the format is correct
  defp parse_task(%{"name" => name, "command" => command, "params" => params, "interval" => interval}),
    do: %__MODULE__.Task{
      name: name,
      command: command,
      params: params,
      interval: interval_to_ms(interval)
    }

  ## In case the file has a invalid entry, we print an error message
  ##   and exit the program
  defp parse_task(invalid_task) do
    write_error("Invalid command > #{inspect(invalid_task)}")
    exit(:shutdown)
  end
  
  ## we gradually merge hour to minute then minute to second and finally second to ms
  defp interval_to_ms(%{"second" => second, "minute" => minute, "hour" => hour}),
    do: interval_to_ms(%{"second" => second, "minute" => minute + hour * 60})

  defp interval_to_ms(%{"second" => second, "minute" => minute}),
    do: interval_to_ms(%{"second" => second + minute * 60})

  defp interval_to_ms(%{"second" => second}),
    do: second * 1000

  # write on stderr
  defp write_error(str), do: IO.write(:standard_error, str)
end

lib/tasks/parser.ex

Ok, let's try our parser. Run iex -S mix to start your program in an Interactive Elixir session and run Omycron.Tasks.Parser.parse:

iex(1)> Omycron.Tasks.Parser.parse("./omc.toml")
Parsed tasks > [%OMC.Tasks.Parser.Task{command: "echo 'hello'", interval: %{"second" => 10}, name: "Hello world"}, %OMC.Tasks.Parser.Task{command: "echo 'im alive'", interval: %{"second" => 3}, name: "Im alive"}]
[
  %Omycron.Tasks.Parser.Task{
    command: "echo 'hello'",
    interval: 10000,
    name: "Hello world"
  },
  %Omycron.Tasks.Parser.Task{
    command: "echo 'im alive'",
    interval: 3000,
    name: "Im alive"
  }
] 
iex(2)>

Alright, it's working well. Finally, we just have to merge this piece of code with the Agent we previously defined:

in tasks/agent.ex, make the following changes to store the result of the parser into our Agent.

 def start_link([config_pathname]) do
    Agent.start_link(fn -> Omycron.Tasks.Parser.parse(config_pathname) end, name: __MODULE__)
 end

lib/tasks/agent.ex

and in application.ex, retrieve the config pathname from using the Config module of Elixir:

children = [
  Omycron.Tasks.Agent.child_spec([Application.fetch_env!(:omycron, :tasks_pathname)])
]

lib/omycron/application.ex

At this point, when you start the program, the tasks are loaded and stored in our agent. You can create a new file tasks.exwhere we will have a small helper to retrieve the list of tasks:

defmodule Omycron.Tasks do
  @moduledoc """
  Helpers for tasks
  """

  @doc """
  Retrive the list of tasks
  """
  def get do
    Agent.get(__MODULE__.Agent, & &1)
  end
end

lib/tasks.ex

Our task runner

Let's wrap this chapter by writing our task runner. The task runner is a GenServer (understand a sub-process) that will send an event to itself every time a task should be executed.

Now everything is clear, create a runner.ex file that looks like the following:

defmodule Omycron.Runner do
  @moduledoc """
  We use a GenServer (a sub-process) that will use the erlang `timer` module to send itself an event that will start the execution of a task
  """
  use GenServer

  @doc """
  Follow the official documentation of GenServer
  """
  def start_link(opts), do: GenServer.start_link(__MODULE__, opts, name: __MODULE__)

  @impl true
  def init(_opts) do
    # small trick to wait for the tasks to load everything
    :timer.send_after(1000, :start)
    {:ok, nil}
  end

  @doc """
  Load the list of tasks and start each of them one by one
  """
  @impl true
  def handle_info(:start, _params) do
    with tasks <- OMC.Tasks.get(),
         :ok <- Enum.each(tasks, &setup_task(&1)) do
      {:noreply, tasks}
    else
      _err -> :error
    end
  end

  @doc """
  Execute a Task and print the result
  """
  @impl true
  def handle_info(
        {:execute, %OMC.Tasks.Parser.Task{name: name, command: command, params: params}},
        tasks
      ) do
    {res, _} = System.cmd(command, params)
    IO.puts("Task #{name} executed successfully:\n#{inspect(res)}\n")
    {:noreply, tasks}
  end

  ## use the `send_interval` function to trigger an event to itself everytime the task should run
  defp setup_task(%OMC.Tasks.Parser.Task{interval_in_ms: interval_in_ms} = task) do
    :timer.send_interval(interval_in_ms, {:execute, task})
  end
end

lib/runner.ex

Now everything is ready, let's add this GenServer to our application's children. Edit application.ex:

  @impl true
  def start(_type, _args) do
    children = [
      # Loads and store the tasks
      Omycron.Tasks.Agent.child_spec([Application.fetch_env!(:omycron, :tasks_pathname)]),
      Omycron.Runner.child_spec([]) # <- add this line
    ]

    opts = [strategy: :one_for_one, name: Omycron.Supervisor]
    Supervisor.start_link(children, opts)
  end

Et voilà! Our CRON is finally working. You can test by running iex -S mix.

I hope you enjoyed this article and stay tuned for the following chapter!

If you have a problem and no one else can help. Maybe you can hire the Kalvad-Team.