Disqus - Latest Comments for waloeiii

Re: Monitoring Everything (Part 2)

waloeiii — Tue, 23 Sep 2014 13:17:19 -0000

What happens when the redis node goes down?

The single point of failure is just moving around. I agree that it is not optimal to spool to local disk regularly, but in any other configuration you are guaranteeing message loss and increasing complexity.

Do you find your redis server to be more reliable than your logstash node? I would argue that is probably a configuration issue - my logstash instances have been extremely reliable once I setup the JVM correctly.

Re: Monitoring Everything (Part 2)

waloeiii — Tue, 19 Aug 2014 16:27:03 -0000

Why use redis? Isn't it just another moving part that might fail. If your logstash indexer goes down, the shipper should spool on its own local disk.

Re: Understanding New Relic Queuing

waloeiii — Thu, 10 Jan 2013 14:22:19 -0000

What about X-Request-Start on the edge load-balancer? X-Queue-Start is generally inserted by the webserver just in front of the application servers, but there is a large amount of time spent in SSL negotiation and various reverse proxies that commonly exist before the application servers.

Re: Nikolay Sturm's Blog: Pragmatic Scalability for Puppet -- How a Cronjob Saved My Puppetmaster

waloeiii — Fri, 09 Dec 2011 12:08:42 -0000

You can just use the splay feature to do this "automagically" for you. Set the splay time for something like 5 minutes, and when all nodes restart, they will check in randomly across the 5 minute window avoiding crushing your puppetmaster.

Re: Edge Rails.info :: ActiveRecord Identity Map

waloeiii — Fri, 22 Apr 2011 19:33:58 -0000

This will probably yield a nice performance benefit when rendering partials in big loops. All of the foo.bar.baz.widget calls will not generate new objects all the way up the chain, which will avoid creating tons of single use objects which then have to be collected.

Re: Starting with Git: Cheat Sheet

waloeiii — Wed, 03 Nov 2010 15:11:18 -0000

If you want to avoid doing git add myfile.txt 8 times if you touched 8 files, it accepts a lot more options.

You could do something like

$ git add *.txt

or if you want everything

$ git add .

Also, you don't have to stage everything in a file you can use interactive add to stage only chunks. Maybe you did a bunch of work in one file but realize it would be easier to manage if it was 3 separate commits.

$ git add -i

Re: OmniTI - Theo Schlossnagle

waloeiii — Thu, 29 Jul 2010 18:51:42 -0000

Tom Kleinpeter is working on the evaluation of gear:

http://www.devdiligence.com/

Re: SimpleGeo - Joe Stump

waloeiii — Thu, 29 Jul 2010 18:35:35 -0000

Cassandra downside: Schema Changes.

Re: How to use Amazon’s S3 web service for Scaling Image Hosting

waloeiii — Wed, 14 Jul 2010 18:17:10 -0000

Instead of managing the cache yourself you could store these in memcache and retrieve them with the nginx memcache module. You could also do something similar with Varnish.

Re: Simple daily git workflow

waloeiii — Tue, 13 Apr 2010 23:32:10 -0000

Recently wrote up how we do this at onehub.com, we wrote some tools around the Lighthouse, Github and Hoptoad APIs to script some of this to make deploying super easy.

http://leigh.onehub.com/bui...

Re: Consuming the Twitter Streaming API

waloeiii — Fri, 19 Mar 2010 15:23:14 -0000

Yajl-Ruby will do this very well.

http://github.com/brianmari...

This is even one of the examples.

Re: #yerdoinitwrong episode 1: logging with syslog

waloeiii — Tue, 22 Sep 2009 20:13:28 -0000

host = <%= hostname %>
index = <%= environment %>
_blacklist = \.(tgz|gz)$

[monitor:///var/log]
disabled = false

[monitor:///data/onehub/shared/log]
disabled = false

[monitor:///data/onehub/shared/pids]
disabled = false

[monitor:///vol/log/mysql-slow.log]
disabled = false

[monitor:///vol/log/mysqld.log]
disabled = false

Re: #yerdoinitwrong episode 1: logging with syslog

waloeiii — Wed, 16 Sep 2009 13:07:25 -0000

I should also point out that you can configure the forwarders to send data to different indexes. Set the index variable in inputs.conf to coincide with the RAILS_ENV on the machine you are deploying to. Searching through splunk defaults to production (I renamed it from main), but if I want to find something on staging just add index=staging to the query. If you aren't sure of the environment you can just search across all indices.

Re: #yerdoinitwrong episode 1: logging with syslog

waloeiii — Wed, 16 Sep 2009 13:04:39 -0000

When using Splunk you don't have to send your logs to syslog, in fact I find it simpler to not do it. Every single one of my machines runs Splunk in Lightweight Forwarder Mode (http://www.splunk.com/base/... and they all forward to a central Splunk server. The Forwarding instances don't require licenses or anything, they will watch whatever files (or folders!) you configure in inputs.conf and then relay that to your central instance. If the central instance goes down, the Forwarders queue messages (up to a determined size) while waiting for the central server to respond. Forwarding with some aggressive logrotate configs keeps my log volume down on the working instances, and I now have 18 months of logs from 17 machines in one nice organized index.

@joegrossberg Splunk 3.x has a live-tail feature that I find is only ~2 seconds off real-time. Splunk 4.x is considerably faster and the regular search is only ~5 seconds off real-time (but no Live Tail in 4.x yet).

Re: Sumo: One-off EC2 Instance Lanching

waloeiii — Wed, 09 Sep 2009 00:39:22 -0000

I think the best is to use as simple of an image as possible and use the EC2 Metadata API to have instances load the appropriate recipes based off of that.

Re: A grand piano for your violin

waloeiii — Sun, 16 Aug 2009 20:03:31 -0000

The query_reviewer (http://github.com/dsboulder... plugin is perhaps the most useful tool for this.

Re: A grand piano for your violin

waloeiii — Sun, 16 Aug 2009 19:58:22 -0000

MySQL does some very weird things with indexes.

http://dev.mysql.com/doc/re...

Its especially important to recognize the order of the columns in a WHERE is significant.
"MySQL cannot use an index if the columns do not form a leftmost prefix of the index"

Using hash notation for the WHERE does not guarantee order (unless you are on Ruby 1.9) so your query could ask for the columns out of order, use the Array conditions syntax if you need to guarantee this or use some Hash that retains order.

When creating composite indexes, its very important to use the more selective element first, for example with a Polymorphic do other_id, other_type since id will be more selective than type (you probably only have a fewer Models than entries.)

On the topic of indexing created_at, why would you when your PK (id typically) is already extremely likely to be in the correct order? There is a small chance if you are looking at the global list, that because of transactions a few ids could be out of order relative to created_at, but its much more common that you are showing me my comments sorted chronologically in which case odds are extremely slim that I've created two that overlap, the id of the comments is more than sufficient and frees the database from maintaining another index.

As another gotcha, MySQL isn't very good at using multiple indexes per table per query, be sure to check your EXPLAIN statements carefully to see query plan.