Comments on: #yerdoinitwrong episode 1: logging with syslog

By: bryanl

bryanl — Tue, 13 Oct 2009 17:46:35 +0000

I'm sure with little toy apps, “tail -f” works. When you outgrow that, you need something better.

By: Tricon

Tricon — Tue, 13 Oct 2009 12:40:52 +0000

It's hard to beat simple tools like tail -f when it comes to observing logs in realtime. I'm more interested in using Splunk to follow up on bug reports, issues, etc. — things I won't hear about in realtime anyways.

By: bryanl

bryanl — Sat, 26 Sep 2009 07:54:54 +0000

Yes. I'm just working out some content, and then we will have a proper
podcast

======

By: chris

chris — Sat, 26 Sep 2009 02:17:54 +0000

Any chance of this becoming a proper podcast so that I can subscribe?

By: waloeiii

waloeiii — Wed, 23 Sep 2009 05:13:28 +0000

host = <%= hostname %>
index = <%= environment %>
_blacklist = .(tgz|gz)$

[monitor:///var/log]
disabled = false

[monitor:///data/onehub/shared/log]
disabled = false

[monitor:///data/onehub/shared/pids]
disabled = false

[monitor:///vol/log/mysql-slow.log]
disabled = false

[monitor:///vol/log/mysqld.log]
disabled = false

By: sant0sk1

sant0sk1 — Wed, 23 Sep 2009 02:09:52 +0000

Hmm, lightweight forwarders seem like a nice way of going about this. Can you provide an example inputs.conf from one of your forwarding instances?

By: joegrossberg

joegrossberg — Wed, 16 Sep 2009 23:18:52 +0000

You don't. But the majority of Rails sites have a different level of traffic and using Splunk — unlike reading your log files — precludes the option of side-by-side debugging via “load the page while simultaneously seeing what appears in your terminal”.

One size does not fit all, and a 10 req/s site is atypical.

By: bryanl

bryanl — Wed, 16 Sep 2009 22:35:12 +0000

How do you have real time when you are doing over 10 requests per second?

By: waloeiii

waloeiii — Wed, 16 Sep 2009 22:07:25 +0000

I should also point out that you can configure the forwarders to send data to different indexes. Set the index variable in inputs.conf to coincide with the RAILS_ENV on the machine you are deploying to. Searching through splunk defaults to production (I renamed it from main), but if I want to find something on staging just add index=staging to the query. If you aren't sure of the environment you can just search across all indices.

By: waloeiii

waloeiii — Wed, 16 Sep 2009 22:04:39 +0000

When using Splunk you don't have to send your logs to syslog, in fact I find it simpler to not do it. Every single one of my machines runs Splunk in Lightweight Forwarder Mode (http://www.splunk.com/base/Documentation/3.3.4/...) and they all forward to a central Splunk server. The Forwarding instances don't require licenses or anything, they will watch whatever files (or folders!) you configure in inputs.conf and then relay that to your central instance. If the central instance goes down, the Forwarders queue messages (up to a determined size) while waiting for the central server to respond. Forwarding with some aggressive logrotate configs keeps my log volume down on the working instances, and I now have 18 months of logs from 17 machines in one nice organized index.

@joegrossberg Splunk 3.x has a live-tail feature that I find is only ~2 seconds off real-time. Splunk 4.x is considerably faster and the regular search is only ~5 seconds off real-time (but no Live Tail in 4.x yet).