In the chaotic world of web development and server management, HTTP logs are our lifeline. They’re supposed to be the reliable friends that keep us informed about traffic, errors, and every hiccup our web applications experience. But here's the kicker: most standard HTTP log solutions are like those friends who conveniently "forget" the crucial details when recounting a story. They don’t capture the body of the request and response, leaving developers blindfolded and stumbling in the dark.

This glaring omission is a developer's worst nightmare. Imagine trying to fix a bug without knowing what data is being sent or received. It's like trying to solve a mystery with half the clues missing. Debugging becomes a Herculean task, filled with guesswork and frustration. Without full visibility into HTTP interactions, understanding the nuances of application behavior is next to impossible, turning even simple issues into complex puzzles.

Monitoring application performance and security is equally hampered. How can you ensure robust security measures or optimize performance when your logs are incomplete? It’s like trying to guard a fortress with half the walls missing. Comprehensive HTTP logs, including the request body, response body, and headers, are not just nice-to-have; they're essential for deep insights, proactive monitoring, and effective troubleshooting.

In this article, we’re going to take a sledgehammer to the limitations of current HTTP logging solutions. We’ll explore the standard logging practices in NGiNX, Apache, and Traefik, highlight the glaring gaps, and introduce a game-changing solution using Gleam and the power of Erlang. Finally, we’ll tackle the elephant in the room: how to store these detailed logs efficiently using Quickwit, a solution that promises to make log storage a breeze. Buckle up, because we’re about to transform your logging game!

HTTP Logs in NGiNX, Apache, and Traefik: The Bad, The Bad, and The Least Worst

Welcome to the wild world of HTTP logs, where the quality of your logs can range from "meh" to "are you even trying?" If you've ever tried to debug an issue with your web application, you know the struggle of sifting through logs that are about as detailed as a child's drawing of a unicorn. Let's dive into the logging practices of three popular web servers: NGiNX, Apache, and Traefik, and see just how bad things can get.

NGiNX: The Log That Almost Could

First up, we have NGiNX. Known for its blazing speed and efficiency, NGiNX is the cool kid on the block. But when it comes to logging, NGiNX is like that friend who gives you a lot of information, but none of it useful.

By default, NGiNX logs tell you who knocked on your door, but not what they said or why they were there. It’s like getting a postcard that says "Greetings from Somewhere!" with no other details. Sure, you can customize the log format to get a bit more information, but capturing the body of requests and responses? Forget it. NGiNX just isn’t that into you.

Apache: The Granddaddy of Vague Logging

Next, we have Apache, the granddaddy of web servers. Apache has been around forever, and it shows. Its logs are like an elderly relative's stories: long-winded and filled with irrelevant details.

Apache logs by default are verbose, but they miss the point entirely. They give you a lot of noise and very little signal. You can spend hours wading through these logs only to realize that the crucial information you need is nowhere to be found. It’s like being handed a dictionary when all you wanted was the definition of one word.

Traefik: The Least Worst of the Bunch

Finally, we come to Traefik, the hipster cousin who’s trying hard to be different. Traefik’s logs are actually not terrible. In fact, compared to NGiNX and Apache, they’re almost...useful. Traefik tries to balance detail and usability, giving you enough information to make sense of what’s going on without drowning you in data.

But let’s not get too excited. While Traefik’s logs are the least worst, they still fall short. Capturing the request and response bodies is still a no-go. It's like Traefik is giving you a pretty good recipe but leaving out the key ingredient. You can guess and improvise, but it’s never quite right.

(Temporary) Conclusion

So, there you have it: NGiNX, Apache, and Traefik, each with their own logging quirks and limitations. While Traefik might win the prize for being the least worst, none of these web servers offer the comprehensive logging we truly need. The missing details in request and response bodies make debugging and monitoring a frustrating game of hide and seek.

But don’t lose hope. In the next part, we’ll dig deeper into what’s missing from these logs and why it’s so crucial for developers to have complete visibility. Stay tuned for the journey to better logging!

The Missing Pieces: Request Bodies, Response Bodies, and Headers

The Phantom of the Logs: Body Requests

First up, body requests. You’d think that knowing what data was sent to your server would be pretty important, right? Well, think again. Most HTTP logs don’t bother with this crucial piece of information. It’s like ordering a pizza and only being told that something was delivered, but not whether it was a pepperoni, a veggie delight, or just an empty box.

Without body requests, you’re left guessing what data your users are sending. Was it a valid form submission, a mischievous attempt to break your app, or just random gibberish? Who knows! Not your logs, that’s for sure.

The Invisible Man: Response Bodies

Next, let’s talk about response bodies. This is what your server sends back to the user, and guess what? Your logs don’t care about it. It’s like having a one-sided conversation where you can only hear what you’re saying and have to guess the other person's reactions. Was your response helpful? Did it even make it to the user? Your logs remain stubbornly silent.

Without response bodies, troubleshooting becomes a maddening exercise. Did your API return the correct data? Was the HTML response rendered correctly? Your logs are about as communicative as a mime in a blackout.

The Ghost in the Machine: Headers

Finally, we come to headers. These little snippets of information are like the post-it notes of HTTP communication, carrying crucial metadata about the request and response. And yet, they’re often missing from our logs. It’s like getting a letter with no return address or date – sure, you have the message, but who sent it and when?

Headers can tell you a lot about what’s going on: content types, authentication tokens, caching directives, and more. Without them, you’re flying blind, hoping that nothing important slipped through the cracks.

The (Missing) Complete Picture

So, what does this all add up to? A lot of frustration and guesswork. Without body requests, response bodies, and headers, your logs are like a puzzle with half the pieces missing. You can see parts of the picture, but the critical details that make everything make sense are nowhere to be found.

As developers, we need full visibility to debug effectively, monitor performance, and ensure security. The missing pieces in standard HTTP logs turn these tasks into Herculean labors. But fear not, dear reader, for in the next part, we’ll introduce a solution that fills in these gaps and transforms your logging game. Stay tuned as we unveil the magic of a proxy in Gleam and the power of Erlang!

The Solution: A Proxy in Gleam - Because Your Logs Deserve Better

Meet Gleam: The New Kid on the Block

First, let’s introduce Gleam. If programming languages were superheroes, Gleam would be the cool new recruit with hidden powers. Gleam is a statically typed language that compiles to Erlang, bringing the best of both worlds: the performance and reliability of Erlang with the safety and clarity of modern type systems.

Gleam lets you write clean, readable code that’s easy to maintain, and it runs on the battle-tested Erlang VM. This means you get the power and concurrency of Erlang without the headaches. Think of it as having a Swiss Army knife that’s also a lightsaber.

Building the Perfect Proxy

So, how do we solve the logging conundrum? By building a proxy in Gleam. This proxy will sit between your existing load balancer and your application, capturing every detail of the HTTP transactions that pass through it. It’s like having a supercharged bouncer at the door, noting down everything that happens.

With this proxy, you can capture body requests, response bodies, and headers. All the missing pieces of the puzzle are now within your grasp. And because it’s built with Gleam, it’s efficient and reliable, capable of handling high loads without breaking a sweat.

The Power of Erlang: Concurrency and Reliability

Why Erlang, you ask? Erlang is the grandmaster of concurrency and fault tolerance. It was designed for telecom systems where downtime is not an option, and it excels at handling numerous simultaneous connections. When you combine this with Gleam’s elegant syntax, you get a solution that’s both powerful and easy to use.

Erlang’s lightweight processes and robust error handling mean your proxy can handle massive amounts of traffic without missing a beat. And if something does go wrong, Erlang’s "let it crash" philosophy ensures that failures are isolated and don’t bring down the whole system.

Could You Use Other Languages?

Sure, you could use other languages to write the proxy – Go, Rust, or even JavaScript might come to mind. But none of them will give you the perfect blend of performance, reliability, and developer happiness that Gleam offers. Other languages might get the job done, but they'll never be as efficient or as elegantly integrated with the Erlang VM. With Gleam, you're not just writing code; you're wielding the power of a language that's designed to make your life easier and your applications more robust.

Why This Solution Rocks

1. Comprehensive Logs: Finally, you’ll have logs that include everything – body requests, response bodies, and headers. No more blind spots.
2. Performance: Thanks to Erlang’s efficient concurrency model, your proxy can handle high volumes of traffic without becoming a bottleneck.
3. Reliability: Erlang’s fault-tolerant design means your logging solution is rock-solid and ready for production.
4. Maintainability: Gleam’s clean, type-safe syntax makes the code easy to read, write, and maintain.

{
  "ClientAddr": "10.101.51.85:46238",
  "ClientHost": "10.101.51.85",
  "ClientPort": "46238",
  "DownstreamStatus": 200,
  "Duration": 13912784,
  "Level": "info",
  "RequestAddr": "10.42.2.48:8080",
  "RequestBody": "",
  "RequestHeaders": {
    "accept": "*/*",
    "connection": "close",
    "host": "10.42.2.48:8080",
    "user-agent": "kube-probe/1.28"
  },
  "RequestMethod": "GET",
  "RequestPath": "/up",
  "ResponseBody": "{\"status\":\"ok\"}",
  "ResponseHeaders": {
    "content-length": "15",
    "content-type": "application/json"
  },
  "StartLocal": 1718379531,
  "StartUTC": "2024-06-14T15:38:51.658749Z"
}

Where to Put All That Glorious Data

The Challenges of Storing HTTP Logs

Let’s get one thing straight: HTTP logs can get big. Really big. And when you’re capturing detailed information like body requests, response bodies, and headers, they grow faster than a teenager on a diet of pizza and soda.

1. Volume: Logs accumulate quickly, and the more detail you capture, the more space you need.
2. Complexity: Managing large datasets isn’t just about having enough storage; it’s about organizing it so you can actually find what you need when you need it.
3. Performance: Querying massive log datasets can be slower than a snail on a sticky bun, especially with traditional storage solutions.

Elasticsearch: The Complex Beast

Elasticsearch is often the go-to solution for log storage, and for good reason. It’s powerful, flexible, and can handle large amounts of data. But it’s also complex and resource-intensive. Setting up and maintaining an Elasticsearch cluster can feel like adopting a pet dragon – it’s impressive, but it needs constant attention and has a tendency to burn everything down when things go wrong.

SQL Databases: The Square Peg in a Round Hole

SQL databases are great for structured data, but when it comes to log storage, they’re like using a butter knife to cut a steak. Sure, it can be done, but it’s not pretty, and it’s certainly not efficient. The rigid schema of SQL databases doesn’t play well with the dynamic and often unstructured nature of log data. Plus, querying large log datasets can make even the most robust SQL server break into a sweat.

Even with advancements like JSONB in PostgreSQL, which allows for semi-structured data storage, using SQL databases for log storage remains suboptimal. JSONB can help by storing JSON documents more efficiently and allowing for indexing, but it still doesn't match the performance and flexibility needed for high-volume log data. The complex queries required to extract meaningful insights from JSONB can become slow and cumbersome as data grows, making it a less-than-ideal choice for massive log datasets.

Enter Quickwit: The Knight in Shining Armor

Now, let’s talk about Quickwit, the log storage solution that’s here to save the day. Quickwit is designed specifically for handling large volumes of log data with speed and efficiency. It’s like the Swiss Army knife of log storage – versatile, powerful, and surprisingly lightweight.

Key Advantages of Quickwit

1. Decoupled Compute and Storage: Quickwit separates compute and storage, meaning you can scale them independently. Need more processing power? Add more compute nodes. Need more storage? Just add more disks. This flexibility ensures that you can efficiently handle growing log volumes without over-provisioning resources.
2. Speed: Thanks to its optimized architecture, Quickwit can handle large volumes of data and return query results faster than you can say “log overload.” Its efficient indexing and retrieval mechanisms mean that you get the information you need, right when you need it.
3. Scalability: Quickwit scales effortlessly, allowing you to store and query terabytes of log data without breaking a sweat. Whether you’re dealing with a few gigabytes or several petabytes, Quickwit adapts to your needs, ensuring smooth performance at any scale.
4. Cost-Effectiveness: Quickwit is designed to be resource-efficient, meaning you can store more data for less money. No more burning through your budget on expensive hardware and maintenance. Its lightweight footprint ensures that you get the best bang for your buck.
5. Ease of Use: Quickwit’s intuitive interface and straightforward setup make it accessible even if you’re not a log storage wizard. You don’t need to be a rocket scientist to get it up and running, and its user-friendly design means you can focus on what really matters – analyzing your logs.

Implementation: Getting Quickwit Up and Running

Setting up Quickwit is a breeze compared to the heavy lifting required for Elasticsearch or trying to shoehorn logs into a SQL database. Here’s a quick overview of how you can get started:

1. Installation: Quickwit’s lightweight installation process gets you up and running in no time.
2. Configuration: Define your log sources and indexing settings. Quickwit’s flexible configuration options let you tailor it to your specific needs.
3. Integration: Hook up your Gleam proxy to send logs to Quickwit. With Quickwit’s robust API, integration is seamless.
4. Querying: Use Quickwit’s powerful querying capabilities to sift through your logs and find the information you need, fast.

Note: We received feedback indicating that we did not provide enough information on setting up Quickwit, so here are some basic details:

The index

version: 0.7

index_id: "raclette-20240604-v3"

index_uri: "s3://quickwit/raclette-20240604-v3"
doc_mapping:
  mode: dynamic
  dynamic_mapping:
    indexed: true
    stored: true
    tokenizer: default
    record: basic
    expand_dots: true
    fast: true
  field_mappings:
    - name: StartLocal
      type: datetime
      input_formats:
        - unix_timestamp
      output_format: unix_timestamp_secs
      fast_precision: seconds
      fast: true
    - name: ClientHost
      type: text
      tokenizer: default
      fast: true
    - name: DownstreamStatus
      type: u64
      fast: true
    - name: Level
      type: text
      tokenizer: default
      fast: true
    - name: Duration
      type: u64
      fast: true
    - name: RequestMethod
      type: text
      tokenizer: default
      fast: true
    - name: RequestPath
      type: text
      tokenizer: default
      fast: true
    - name: RequestHeaders
      type: json
      tokenizer: default
      fast: true
    - name: ResponseHeaders
      type: json
      tokenizer: default
      fast: true
    - name: RequestBody
      type: text
      tokenizer: default
      fast: true
    - name: ResponseBody
      type: text
      tokenizer: default
      fast: true
  timestamp_field: StartLocal

search_settings:
  default_search_fields:
    [
      ClientHost,
      DownstreamStatus,
      Duration,
      RequestMethod,
      RequestPath,
      request_body,
      response_body,
    ]

retention:
  period: 360 days
  schedule: daily

Simply make a curl request to the Quickwit API, and you're all set to log!

curl -XPOST http://localhost:7280/api/v1/indexes -H "content-type: application/yaml" --data-binary @raclette-20240604-v3.yaml

Then you are ready to query!

Wrapping Up Our Logging Adventure

And there you have it, folks! We've journeyed through the treacherous lands of HTTP logging, faced the monsters of incomplete logs, and emerged victorious with a solution that's as powerful as it is elegant. Let's take a moment to bask in the glory of our newfound logging prowess.

The Struggle Was Real

We started our adventure in the dark ages of HTTP logs, where NGiNX, Apache, and Traefik showed us just how "meh" logging can be. Sure, Traefik was the least worst of the bunch, but let's be honest, that's like being the least soggy french fry in a batch that's been sitting out too long.

Missing Pieces: The Plot Thickens

Then we ventured into the shadowy realm of missing log pieces – body requests, response bodies, and headers. It was like trying to solve a murder mystery with half the clues missing and a detective who prefers naps over solving crimes. But we knew there had to be a better way.

Gleam and Erlang to the Rescue

Enter our heroes: Gleam and Erlang, the dynamic duo we didn't know we needed. They swooped in with their clean syntax and robust performance, setting up a proxy that could capture every tiny detail we craved. Suddenly, our logs went from being cryptic hieroglyphs to a clear, detailed narrative.

Quickwit: The Storage Solution We've Been Dreaming Of

But wait, there's more! Capturing detailed logs is one thing; storing them is another beast entirely. We discovered that traditional storage solutions like Elasticsearch and SQL databases were like trying to store an elephant in a broom closet. Enter Quickwit, our knight in shining armor. With its decoupled compute and storage, blazing speed, and effortless scalability, Quickwit turned our log storage nightmares into a sweet dream of efficiency.

The Final Countdown

So, what have we learned? Comprehensive logging isn't just a pipe dream. With the right tools, it can be a reality. We’ve gone from the dark ages to the renaissance of HTTP logging. No more guessing games, no more blindfolded debugging. Just clear, detailed, and beautifully stored logs.

And here's the cherry on top: our Gleam proxy solution will soon be released as an open-source project! Stay tuned for the release and join us in revolutionizing the world of HTTP logging.

Now, armed with Gleam, Erlang, and Quickwit, you’re ready to conquer the world of HTTP logs like the superhero you are. Go forth and log all the things, with the confidence that you’ve got the best tools in the business.

Thanks for joining us on this wild ride. May your logs be detailed, your queries swift, and your storage solutions elegant. Until next time, happy logging! 🚀