Disqus - Latest Comments for danmil

Re: Hey, how's dumbo running a M/R combine? Uh, well...

danmil — Wed, 14 Jan 2009 15:17:56 -0000

Klaas, that's *awesome*. I just took a look at GitHub this morning and saw how much work you'd been doing lately. Thanks a ton. And, yes, the newlines are nice to see ;-)

Seriously, though, the piecewise-combine is a big, big win for us. Thanks again (and thanks for Dumbo in general, which is lovely stuff).

I look forward to integrating this soon (at which point I'll hopefully be able to submit a patch which allows you to use -reducer aggegrate, which I've found pretty helpful).

Re: The Absolute Most Useful Thing I've Done At My Job So Far

danmil — Wed, 14 Jan 2009 11:35:14 -0000

Thanks ;-)

I'd love to open-source it (and I would strongly suspect Lookery would be fine with that), but it's pretty tightly linked to our overall scheduling-of-jobs system. Which, I don't know if people want/need that or not. And which would need a bit more love before it could be used in general .

Re: Hey, how's dumbo running a M/R combine? Uh, well...

danmil — Mon, 29 Dec 2008 15:01:04 -0000

Excellent point -- there's no particularly good reason that dumbo/poop materialize the entire list. It doesn't need to sort the entire thing -- just whatever subset its about to combine.

Thanks for pointing that out -- I hadn't realized that the Java side combines as it goes, so to speak

-Dan M

Re: Hey, how's dumbo running a M/R combine? Uh, well...

danmil — Wed, 24 Dec 2008 10:32:42 -0000

Christopher: thanks much for the link. From a quick look, good stuff.

-Dan M

Re: Erlang, I Love You, Erlang, I Hate You

danmil — Tue, 04 Nov 2008 09:16:21 -0000

Thanks for the feedback -- much appreciated.

Short answer for now: we're not doing any data analysis in Erlang, and I'd be somewhat surprised if that changes. Erlang is great for writing the demanding server bits, but Hadoop with Python for the M/R jobs looks like a winner for the data analysis part of the puzzle.

Longer answer: at some point soon-ish I'll try to write up a sketch of how the pieces fit together.

-Dan M