Bryan Says...

Rants, opinions, and other stuff I find interesting.

#yerdoinitwrong episode 1: logging with syslog

We always give all of our best work to development. We create code we are proud to attach our name to, and then we throw it on production. When it gets to production, there are going to problems, and the first thing you are going to do is head to your logs. So why aren’t we taking more steps to have easier accessible logs?

Log management in Ruby on Rails isn’t very good out of the box. When your application starts getting any level of decent traffic, tailing the log isn’t very effective. Searching the log is even harder, and correlating data is pretty much out of the question. The good thing is that is pretty much a solved problem, so let’s review one potential solution.

With Ruby on Rails, you can easily swap our your logger. So instead of the instead of the standard Ruby logger, we’re going to swap in SyslogLogger gem from seattle.rb. Once you have either installed the gem or configured Ruby on Rails to install the gem, you can configure it in your environment.rb. (or one of your environment specific configuration files)

RAILS_DEFAULT_LOGGER = SyslogLogger.new
Rails::Initializer.run do |config|
  ...
  config.logger = RAILS_DEFAULT_LOGGER
end

Now Rails will send the logs to syslog. Next, you’ll want to configure syslog to send all your logs to a remote server we’ll talk about later

*.*	@remote.server

Make sure to restart your syslog server after making these changes.

Now why would we want to send out logs to a remote server? Because central log management is the bees knees of course.

When it comes to log management, nothing beats Splunk. Nothing. So that means we should send our logs to our Splunk server. The Splunk guides make configuring this easy, and their documentation is superb.

Now that you have Splunk configured, you should be receiving logs. The amount of options it gives you is absolutely breath taking, so I advise you to spend some time learning it.

Now if I’m doing it wrong let me know in the comments. Word.

  • tcrawley

    Bryan: thanks for doing this! This is the answer to a problem I was planning on solving in the next couple of weeks. I'm looking forward to seeing what else I'm doing wrong.

  • joegrossberg

    This might be outdated, but I think these points are worth mentioning:

    * Splunk can also take in logs from non-Rails apps (e.g. have your Rails, Java and Apache logs all in one place) and multiple projects
    * Splunk can be overly strict about the logging format it expects :/
    * Splunk is not real-time, unlike tail -f

  • waloeiii

    When using Splunk you don't have to send your logs to syslog, in fact I find it simpler to not do it. Every single one of my machines runs Splunk in Lightweight Forwarder Mode (http://www.splunk.com/base/Documentation/3.3.4/…) and they all forward to a central Splunk server. The Forwarding instances don't require licenses or anything, they will watch whatever files (or folders!) you configure in inputs.conf and then relay that to your central instance. If the central instance goes down, the Forwarders queue messages (up to a determined size) while waiting for the central server to respond. Forwarding with some aggressive logrotate configs keeps my log volume down on the working instances, and I now have 18 months of logs from 17 machines in one nice organized index.

    @joegrossberg Splunk 3.x has a live-tail feature that I find is only ~2 seconds off real-time. Splunk 4.x is considerably faster and the regular search is only ~5 seconds off real-time (but no Live Tail in 4.x yet).

  • waloeiii

    I should also point out that you can configure the forwarders to send data to different indexes. Set the index variable in inputs.conf to coincide with the RAILS_ENV on the machine you are deploying to. Searching through splunk defaults to production (I renamed it from main), but if I want to find something on staging just add index=staging to the query. If you aren't sure of the environment you can just search across all indices.

  • http://smartic.us bryanl

    How do you have real time when you are doing over 10 requests per second?

  • joegrossberg

    You don't. But the majority of Rails sites have a different level of traffic and using Splunk — unlike reading your log files — precludes the option of side-by-side debugging via “load the page while simultaneously seeing what appears in your terminal”.

    One size does not fit all, and a 10 req/s site is atypical.

  • sant0sk1

    Hmm, lightweight forwarders seem like a nice way of going about this. Can you provide an example inputs.conf from one of your forwarding instances?

  • waloeiii

    host = <%= hostname %>
    index = <%= environment %>
    _blacklist = .(tgz|gz)$

    [monitor:///var/log]
    disabled = false

    [monitor:///data/onehub/shared/log]
    disabled = false

    [monitor:///data/onehub/shared/pids]
    disabled = false

    [monitor:///vol/log/mysql-slow.log]
    disabled = false

    [monitor:///vol/log/mysqld.log]
    disabled = false

  • chris

    Any chance of this becoming a proper podcast so that I can subscribe?

  • http://smartic.us bryanl

    Yes. I'm just working out some content, and then we will have a proper
    podcast

    ======

  • http://triconium.com Tricon

    It's hard to beat simple tools like tail -f when it comes to observing logs in realtime. I'm more interested in using Splunk to follow up on bug reports, issues, etc. — things I won't hear about in realtime anyways.

  • http://smartic.us bryanl

    I'm sure with little toy apps, “tail -f” works. When you outgrow that, you need something better.