If you follow us on LinkedIn, you should have seen that we are hiring a lot of people these days, especially in one department that is not the most active on the hiring part: Kalvad's Infrastructure Team (KIT™).

The KIT™ has currently 2 members with me as the manager, and it was enough for so long, thanks to all our processes! But nowadays, we need to hire more, as I need to be out of operations in order to work more on the R&D side.

The Jobs

Junior Position (0-2 years of working experience)

Responsibilities

• Learn from our Senior Engineers

Qualifications

• BA/BS degree in Computer Science, Mathematics or related technical field, or equivalent practical experience.
• Experience writing software in one or more languages such as Python, C, Erlang, Elixir, Rust, Ruby, Go, and/or similar.
• Experience with Linux or *BSD.
• Arabic spoken and written is a plus.

Intermediate position (2-3 years of working experience)

Responsibilities

• Learn from our Senior Engineers

Qualifications

• BA/BS degree in Computer Science, Mathematics or related technical field, or equivalent practical experience.
• Experience writing software in one or more languages such as Python, C, Erlang, Elixir, Rust, Ruby, Go, and/or similar.
• Experience with Linux or *BSD.
• Arabic spoken and written is a plus.
• Experience managing 100+ servers.
• Experience with Network (BGP, IPSec, Wireguard, ...)
• Experience with internet protocols (HTTP, DNS, ...)
• Experience with Ansible

As you can see, we don't ask for a lot!

Hiring Philosophy at Kalvad!

Usually, at Kalvad, we prefer to hire Juniors. Why? Because they don't have yet the bad habits! So how do we hire? How do we find our candidates?

Unfortunately, the local universities tend to offer a curriculum that doesn't match our requirements. We are actively partnering with some schools to change that - so that we can tap into the local talent pool in the future. However, for the time being, we set our sights on LinkedIn.

Furthermore, especially for jobs like this, the most important part is the training part: we shared a lot of our tricks inside this blog, but we need to teach them to our new recruits, and it can take between 6 months to a year to do this.

Unfortunately, we can only do this if some basic knowledge has been learned already. And that's what I'm afraid of.

SRE/DevOps/Sysadmin interviews!

At Kalvad, we like to criticize technologies (like Kubernetes, Docker, PHP, Ruby, BlockChain), but what people don't know, is that we were using these technologies at some point, so we can easily interview people on it!

As I'm in charge of the KIT™, I manage the first round of interviews myself, and I will share today how to succeed in an interview with us!

A traditional interview is 30 minutes: 2 minutes to introduce myself and the company, 2 minutes for the candidate to introduce himself, then we are doing through multiple steps, starting with a process that I call the gravedigging!

Gravedigging

I will go through the CV, and every time a technology is mentioned, I will ask a question about it!

I will write some technologies that I see, and what we expect as an answer!

Docker/Linux Containers

If your answer is less than what is written below this, you should not put Docker in your CV.

A container is a Linux process that is using some kernel capabilities which are:

• Linux Namespaces
• Linux CGroups (v1 or v2)

DevOps

DevOps is a culture that stops the infamous process of Devs dropping bombs on Ops: it means collaborations, training, planning, and anticipation!

Prometheus

Prometheus is not a Time series database! It's a monitoring and alerting tool, which has the capabilities to store some metrics! (If you want a real TSDB, check Warp 10).

Operational Knowledge Evaluation

After being sure that they know the technologies (I already stop 50% of the interviews at that point!), I will start asking more questions about it, and here is my problem! People don't know the technologies. One of my favorite questions as some Prometheus "experts" is how do you make Prometheus highly available! Or I will ask a Kubernetes expert which Ingress he/she is recommending, and to my surprise, 99% of them are going to say NGiNX! Even if NGiNX is an amazing software, the open-source version is too limited to do this!

Real-life test

As part of our process, we like people to show us some real-world examples of what they can do! Here is the test that we were using until today (I will have to find another one!)

Introduction

Kalvad is facing an issue: we don't like Google Analytics, because Google is evil, but we need to set up an alternative that can track our users’ behaviors.

Solution
We would like to deploy Umami, unfortunately, Soufiane or I have no time to review it, and to prepare a plan to deploy it.

Task
Your target is to provide documentation in order to deploy Umami, in a safe way, in order to set it up on our website. A VM is going to be provided for you to build the base configuration. The VM is based on Ubuntu 20.04. We are also going to provide a DNS entry to this VM!
The output of this task should be:
- A VM with Umami deployed
- A git repository on GitHub, with full documentation on how to do it!

Please note that it should be a complete guide in order to go to production!

Good Luck

This test is perfect for us, as Umami is a very simple software, which does not require a lot of 3rd party software (only PostgreSQL or MySQL).

But people don't deploy it properly! Here are some common mistakes we saw:

• MySQL opens with remote access and no password on the root account (thanks to some crazy MySQL Docker containers)!
• Keeping the username/password of the doc!
• Forgot to make it start when the server reboots
• Run it as root
• No backup procedure documented!

Out of the last 56 interviews, 28 got the test, 8 submitted the test, 1 got it right!

Pauperization of the job

One person that I respect a lot said this amazing phrase:

In SF, a dog with a Kubernetes t-shirt could get a devops job!
Quentin Adam, CEO of Clever Cloud

Even if it is funny, it makes me feel sad, as I see the results of this problem in my everyday job!

We ask our sysadmins to run ArchLinux on their work computer, we expect them to compile their own software, compile their own Linux kernel, understand file systems, network protocols like BGP, IPSec, learn programming languages (like Zig, the most amazing language of the last 3 years!).

But we are not matching the market! It's extremely hard to find people to join us, as most sysadmins are becoming one of these 2 jokes:

• They are getting all possible certifications as proof that they know the job, but let's be clear: having a certification, that you can just buy most of the time, is not making you operational ready for these platforms/software! A Kubernetes certified (CKA) person did not know what was a container, or that Kubernetes was written in Go!
• They are becoming button-pushers: they take a manual (or a Helm chart), and execute the job, as a machine, without thinking about the implications! They think that everything is magic, that the cloud is just a magic powder that you can spread everywhere to make your company DevOps compliant!

So we have a choice between people with the same mindset as the traders in the 80s or some cloud supermarket cashier!

Just to be clear, the traders are not what they were, they lost a lot during the last 30-40 years! And the cashier is getting replaced by automated checkout! So which future do we have for these people? None!

Who is to blame?

Cloud companies

I'm tired of hearing that because we don't push people to use Kubernetes in production, we are an old company! One guy tried to humiliate me after a conference because I was complaining about containers! These companies should stop pushing people to use/learn technologies because it's hype, and focus more on their customers' needs and delivering according to business needs, not some sales targets!

Bootcamps

"Become a DevOps in 3 months" is something that I see too much! I will put it in an easy way: to become a good sysadmin, it takes years! So, boot camps, stop selling your low-quality product!

People

How many times do I see the kind of article like: "Get all AWS certifications gives you a +50% salary!". Stop believing these marketing people, stop getting certifications! Just stop it! I cannot stand anymore talking with an AWS expert not able to explain how ELB is working! Furthermore, if you say that you are an expert, be an expert!

Managers

Because they thought that Sysadmins are not needed anymore and that a dev who knows how to ssh could manage the entire Ops for a company!

Us, sysadmins

Because we let the wolf guard the sheep! It's time for us to take back our job, and fight to make the IT world a better place!

Conclusion

We cannot let the situation become more dramatic, we are seeing many companies losing money or closing because the sysadmin part is not covered! At Kalvad, we are planning for next January, to launch a Linux User Group in Dubai, and to help our friend Sven to relaunch DevOps Dubai (which should not be AWS Dubai)! But we cannot do this only by ourselves, so if you have more ideas to improve the situation, contact us, or just do it!

If you have a problem and no one else can help. Maybe you can hire the Kalvad-Team.