Disqus - Latest Comments for jmason

Re: Scaling Analytics at Amplitude

jmason — Mon, 31 Aug 2015 10:02:06 -0000

Thanks for listing costs as one of the driving requirements here -- so often, large-scale architectural discussion omits this requirement, bizarrely!

Re: iosnoop For Linux

jmason — Thu, 17 Jul 2014 06:48:11 -0000

ah, apologies. copy and paste strikes again. cheers ;)

Re: iosnoop For Linux

jmason — Thu, 17 Jul 2014 05:25:03 -0000

Quick portability glitch -- I had to make this change to get it running on Ubuntu 12.04: https://gist.github.com/jma...

Re: Part 2: Lessons learned tuning TCP and Nginx in EC2

jmason — Wed, 12 Mar 2014 12:15:40 -0000

fwiw, we've had great results with SmartStack (albeit on low-rps internal services so far).

BTW, the 'raw logs from ELB' point has been implemented: http://docs.aws.amazon.com/...

Finally, another upside of ELB which you hadn't mentioned was the price -- effectively free ;) This compares pretty well to the cost of provisioning a fleet of nginx/haproxy hosts...

Re: Part 2: Lessons learned tuning TCP and Nginx in EC2

jmason — Wed, 12 Mar 2014 06:55:16 -0000

'each their cloud service is EC2 + appropriate Open Source tool'

just to respond to this -- speaking as an ex-employee of Amazon, this is most definitely not accurate.

Re: Part 2: Lessons learned tuning TCP and Nginx in EC2

jmason — Tue, 18 Feb 2014 17:58:09 -0000

Yep, we did -- after quite a bit of back-and-forth, they offered to effectively turn off downscaling on the ELB which was suffering the issue. As far as I know, it hasn't recurred since then.

Re: Part 2: Lessons learned tuning TCP and Nginx in EC2

jmason — Tue, 18 Feb 2014 09:24:33 -0000

take a look at the graph in the "Next Steps" section.

Re: Part 2: Lessons learned tuning TCP and Nginx in EC2

jmason — Fri, 14 Feb 2014 13:10:23 -0000

Since you have a very clear day/night cycle, I'd be curious if you've seen cases where your servers are marked as unhealthy during times when the ELB is scaling up or scaling down -- we ran into this, and it appears to be an ELB bug. Also, igor47, SmartStack looks very promising for internal LBs...

Re: A Gotcha When Using ZooKeeper Ephemeral Nodes

jmason — Fri, 27 Sep 2013 08:23:56 -0000

hmm. looking into this, I note the following in the ZK javadocs:

http://zookeeper.apache.org..., byte[], java.util.List, org.apache.zookeeper.CreateMode) : 'If a node with the same actual path already exists in the ZooKeeper, a KeeperException with error code KeeperException.NodeExists will be thrown.'

shouldn't that be the case in this scenario?

Re: CloudFlare, PRISM, and Securing SSL Ciphers

jmason — Mon, 17 Jun 2013 06:12:15 -0000

Very interesting post.

BTW one possible reason that some companies are still stuck with 1024-bit RSA keys on their HTTPS endpoints is hardware: up until recently, hardware load balancers like the Citrix NetScalers had a hard time dealing with long key lengths at scale.

Re: Communication Costs in Real-World Networks | Peter Bailis

jmason — Fri, 24 May 2013 12:24:45 -0000

Great data! As Malte noted, there will probably have been heavy impact in the higher percentiles from other VM tenants. It might be worth trying a test using m2.4xlarge instances, if the budget can stretch. ;)

You'd still see impact from rack contention from other hosts on the same rack, though -- maybe using cluster compute instances would be useful to get that out of the picture too.

Mind you -- both of those are a realistic factor for "real-world" EC2-based applications, so they are worth modelling.

Finally, what protocol was used? ICMP is prioritised differently from TCP or UDP traffic.

Re: PopCap shutdown-another EA casualty?

jmason — Tue, 21 Aug 2012 17:16:32 -0000

Sorry to hear this. I guess some good news is that the Dublin tech scene seems to have no shortage of open positions going. Our team at Amazon Dublin, Network Monitoring, is no exception -- we are looking for both software engineers and ops people, and there's many more teams here doing likewise. Check out the rather extensive list of jobs at the Dublin office: http://preview.tinyurl.com/... .

Good luck to all going through this -- I've been there and it's no fun...

Re: Neon Indian – ‘Polish Girl’

jmason — Fri, 05 Aug 2011 18:59:13 -0000

hey, excellent! I have a Bleep Labs Thingamagoop right here ;)

Re: Some Notes on Distributed Key Stores

jmason — Tue, 21 Apr 2009 05:04:07 -0000

Thanks -- great writeup! I hadn't paid much attention to Tokyo Tyrant, but I'll be changing that.

Were you looking for any backup capability? can you snapshot the state of the Tokyo Cabinet store to take a backup of that? or are you just relying on doing that via EBS?

(also: S3 as a k-v store: slooooow)

Re: Virgin America’s Crappy Online User Experience

jmason — Tue, 07 Apr 2009 05:07:43 -0000

when I run into one of those 1024-character-limit "feedback" forms, I tend to write as long as I like -- then cut and paste multiple feedback messages consisting of ~1024-char chunks. I consider it a form of civil disobedience against their crappy user interface. ;) It's almost definitely all going to a single person anyway.

Re: random($foo): Additional Personal Storage

jmason — Wed, 31 Dec 2008 06:40:39 -0000

yeah, I think it's all down to the timing -- how long will it take for the prices to drop enough. From what I've read it seems there's no physical reason that's going to stop them becoming cheaper than spindles in the long run. The sooner, the better...

Re: random($foo): Additional Personal Storage

jmason — Tue, 30 Dec 2008 21:11:55 -0000

check this out: http://perfcap.blogspot.com...

'In 2009 SSD's will be faster for read, faster for write, faster for sequential and much much faster for random access, more reliable, more durable, lower power, higher capacity, than discs. Give it another year or so and they will be cheaper as well.'

Adrian's a legendary Sun performance guy -- imo he probably knows what he's on about... ;)

Re: random($foo): Additional Personal Storage

jmason — Tue, 30 Dec 2008 06:40:02 -0000

Hey Leonard --

I spent a while using a bunch of USB enclosures with a Linux laptop, backing stuff up over the wifi network using rsync onto them. it was pretty crappy; about once a week or so, one of the enclosures would screw up and have to be manually force-umounted, power cycled, fscked and remounted. Very unpleasant -- and risky of course :(

I eventually scavenged some pretty recent desktop hardware and took all the disks out of the enclosures and put them directly into the case as ATA devices. it's been rock solid since then. IIRC, sdparm supports spindown too although I haven't tried it recently.

My conclusion: cheap USB enclosures = not safe at all and best avoided for anything but "toy" use....

(PS: I like yahooza's idea of rsyncing to the ext disk as a secondary backup of a "main" backup. seems safe enough since you still have the primary backup working fine anyway, even if that one fucks up for a few days...)

(PPS: if you can wait a few months before buying new disks, btw, I've been hearing that SSDs are about to just blow all spinning rust out of the water for pretty much any purpose -- cheap, fast, high-bandwidth, reliable, etc.)

Re: Do Your Cloud Applications Need To Be Elastic?

jmason — Wed, 26 Nov 2008 05:18:58 -0000

Hey James -- sorry for the delay, disqus didn't let me know there was a reply ;)

We're an early-stage startup, so no, we had no existing infrastructure bar a few colo servers which could be entirely replaced by EC2 instances.

Being able to use small numbers of servers at a time and be billed for just the CPU time used is great for us. Of course, we also have inelastic parts of the infrastructure that could be hosted elsewhere at a colo for less cost, and personally, I would probably have done this given the choice; but mgmt were happier just to use EC2 as widely as possible, despite the additional costs, since it keeps things simpler.

For example, we also use S3 heavily, and EC2-to-S3 traffic is extremely fast and cheap compared to external-to-S3; so that's proving to be an effective point of lock-in. (We don't mind.)

Regarding locally-hosted servers on our own infrastructure -- that was simply out. There's a massive amount of investment that would be required to get a sufficiently-reliable infrastructure set up, and a good deal of coding and config to get an EC2ish elastic server provisioning system going on that. Further down the line, that might be appropriate, but not yet... it's a lot easier to let Amazon do it for us ;)

There's definitely a point where we *could* migrate to our own infrastructure in the future, and open source apps like Eucalyptus make that more viable -- but in my opinion it's very far off indeed...

Re: Tolling the bell for the gatekeepers

jmason — Tue, 08 Jul 2008 09:09:46 -0000

Strongly agreed!

I tend to rely on my social network, summarised and aggregated using a tool I wrote called Spicylinks: http://taint.org/2006/09/06... . That's the way to do it...

Re: random($foo): Rearchitecting Twitter, Back of the Napkin

jmason — Thu, 29 May 2008 05:21:47 -0000

hi Leonard! I like it.

What about the issue of Scoble's 25,000 followers? Every time he updates, you'd have to append to 25k mboxes. that strikes me as a hard challenge for this design.

Re: We need a Wikipedia for data - Bret Taylor's blog

jmason — Wed, 09 Apr 2008 06:08:55 -0000

hi -- take a look at CKAN.net; it seems vaguely related.