Disqus - Latest Comments for schmichael

Re: Alex Gaynor -- About Python 3

Michael Schurter — Fri, 03 Jan 2014 18:49:11 -0000

I guess almost all of the planned changes made it in: http://www.python.org/dev/p...

I'm just sad there wasn't a more comprehensive stdlib overhaul.

Re: Alex Gaynor -- About Python 3

Michael Schurter — Tue, 31 Dec 2013 12:22:59 -0000

I couldn't agree more! I'm very sad the stdlib reorg didn't make it into Py3k. That could have provided a clear user visible benefit.

I kind of wish Python 4 could happen in the next few years that would focus on reorging the stdlib and porting existing IO modules to Tulip. Now *that* would be a reason for users to work hard at transitioning.

Re: Alex Gaynor -- About Python 3

Michael Schurter — Mon, 30 Dec 2013 16:11:25 -0000

Thank goodness! I just saw tulip on pypi with 2.6+ support.

Re: Alex Gaynor -- About Python 3

Michael Schurter — Mon, 30 Dec 2013 14:40:45 -0000

-1 to Python 2.8.

+1 to PyPy going Python 3 only. Python 3 has always needed more carrot, and since porting to PyPy is often nontrivial anyway it seems like the perfect fit.

+1 to builtin pip+venv for Py3k-only for the same reasons.

I know it seems coercive, but as long as core development resources are evenly split between 2 & 3 there's no reason to leave 2.

I'm just sad Tulip isn't Py3k only. It supports 2.6 for fucks sake!

As long as Python's BDFL's pet projects support antiquated versions of Python, I can't imagine the rest of us switching.

Re: signalfd

Michael Schurter — Tue, 16 Jul 2013 00:13:53 -0000

As I said in the opening paragraph: "This article covers signalfd, a system call only available on Linux. If anyone knows of an equivalent for OSX or BSDs,* please let me know."

I'm only personally interested in Linux, but please let me know if you find out anything about OSX or BSDs.

Re: Building Python Packages with Maven

Michael Schurter — Thu, 09 May 2013 16:06:52 -0000

Maven is definitely big and complex, but I can't think of a sophisticated build system that isn't. Big XML files aren't my favorite to deal with, but it's a style thing more than a technical impediment.

Re: MmStats in Scripts

Michael Schurter — Fri, 05 Oct 2012 12:30:36 -0000

Not yet, but now that 3.3 is out I should really try. 2.6, 2.7, and 3.3 support would be ideal.

Re: Building Python 2.6.8 on Ubuntu 12.04

Michael Schurter — Tue, 29 May 2012 19:18:07 -0000

Python 2.6 is only receiving security updates sadly. I think there are PPAs to get Python 2.6 packages for Ubuntu 12.04 as well, but I prefer to build it myself (when possible).

Re: Introspecting Running Python Processes

Michael Schurter — Fri, 09 Mar 2012 21:00:40 -0000

Please drop by the Python introspection sprints next week either at PyCon or remotely. We're hoping to get a lot of work done on mmstats integration with Django, better tools for working with metrics (a Graphite bridge perhaps?), and anything else Python introspection/instrumentation related. Feel free to post any ideas, questions, or comments on the list: https://groups.google.com/g...

Re: Introspecting Running Python Processes

Michael Schurter — Fri, 09 Mar 2012 20:55:24 -0000

Coda Hale's talk:https://codahale.com/codeco...

Re: Introspecting Running Python Processes

Michael Schurter — Fri, 09 Mar 2012 20:54:38 -0000

https://github.com/robotada...

Re: Failing with MongoDB

Michael Schurter — Wed, 08 Feb 2012 12:41:10 -0000

Great tip. We didn't adjust the dirty_ratio as far as I know, so that probably caused unneeded IO. That being said I don't remember iowait being an issue; most of our issues were with the global write lock and complexities of sharding.

Re: Failing with MongoDB

Michael Schurter — Mon, 06 Feb 2012 19:04:48 -0000

4x RAID 0+1 SAS drives for the data files, but I think you're missing the point: our dataset fit in memory, we had syncing turned off, and we didn't use journaling - the disks should never be touched.

Re: Failing with MongoDB

Michael Schurter — Tue, 10 Jan 2012 18:17:53 -0000

Because we did not need SQL, transactions, relations, a write ahead log, and many of the other features provided by relational databases. We thought we were picking the right tool for the job. Had we known then what we know now we would have picked a different solution, but I doubt it would ever be a relational database simply because it offers a completely different feature set than we required.

Re: Failing with MongoDB

Michael Schurter — Wed, 09 Nov 2011 13:58:45 -0000

Yes, we are using a custom _id value: a uuid4 encoded as a hex string. It's random and therefore distributes evenly. We let mongos figure out the partitions and it seems quite evenly balanced.

Re: Login - 10gen Confluence

Michael Schurter — Mon, 07 Nov 2011 17:26:37 -0000

It would be very helpful to include how to actually get and set the set member priorities on this page.

Re: Failing with MongoDB

Michael Schurter — Mon, 07 Nov 2011 15:41:47 -0000

If I have to "divine hidden meaning" from my database, it has failed me. This is a failure condition. Nothing is perfect, and I fully expect to have to do hard work to learn a system and use it properly. I feel like I have done these things and applied best practices. "Divining hidden meaning" is not something I ever want to have to do with a data system. It implies a level of unpredictability, complexity, and lack of documentation that's unacceptable for my use case.

I'm not advocating everyone throw MongoDB under a bus and migrate all of their apps and data to something else. I'm relaying my experiences in hopes of helping others make educated decisions. No one should have to "divine hidden meaning."

If the answer to my issues are "just wait until release N+1", that's great advice for future generations but doesn't really do a thing to solve my current woes.

Support contracts do seem like a great idea. I appreciate 10gen's eagerness to help. That's very commendable. I'm sure bringing them in earlier would have helped. That being said, I'm confident I'm applying best practices. This use case was not at all complex. If I'm wrong and it takes a support contract to adequately fulfill this straightforward use case, I don't think this is a product I want to be using. YMMV

Re: Failing with MongoDB

Michael Schurter — Mon, 07 Nov 2011 13:38:58 -0000

Added an update. tl;dr Everything in this post related to replica sets and sharding uses 2.0.0.

Re: Failing with MongoDB

Michael Schurter — Mon, 07 Nov 2011 13:38:02 -0000

The schema is half a dozen simple fields with an index on _id and another field. No lists or sub-objects/docuemnts.

The vast majority of our operations are to get or update a specific _id with periodic updates to the secondary index. There are no full collection scans or index misses.

The entire dataset and indexes fit in memory. We have journaling and fsyncing turned off as replication is sufficient durability for this ephemeral dataset.

The update load in conjunction with the global write lock just means we're limited around 3k updates/s per core. Since the write lock is per mongod, that means we've had to shard just to take advantage of multiple cores on a single server.

Re: Failing with MongoDB

Michael Schurter — Mon, 07 Nov 2011 13:33:27 -0000

See the last line of my post: "So we’re trying to migrate off of MongoDB to a custom service better suited for this dataset ASAP."

I'm not involved in this development effort, so you probably won't hear much more from me on the migration effort.

Re: Failing with MongoDB

Michael Schurter — Mon, 07 Nov 2011 13:32:35 -0000

Just wanted to point out this is precisely the case. That being said we've had issues with other distributed databases with a similar sized dataset. Distributed systems are hard.

Re: Failing with MongoDB

Michael Schurter — Mon, 07 Nov 2011 13:31:18 -0000

While many organizations happily use MySQL as a NoSQL database, we have not pursued that avenue internally at this point. Currently our efforts have mostly focused on PostgreSQL, Cassandra, HBase, and, obviously, MongoDB (throw in some Redis, Memcached, and S3 as well).

There's only so much time you can spend experimenting with databases and migrating datasets and application code.

Re: Failing with MongoDB

Michael Schurter — Mon, 07 Nov 2011 13:29:09 -0000

This is not a question with a simple easy answer. The usual suspects are: HBase, Cassandra, Riak, Redis, etc. Understand your workload as best you can, then pick the database best suited for that use case.

Re: Failing with MongoDB

Michael Schurter — Mon, 07 Nov 2011 13:27:20 -0000

I'm glad MongoDB is working for your use case.

Re: Failing with MongoDB

Michael Schurter — Mon, 07 Nov 2011 13:25:41 -0000

That is precisely what we did to utilize more cores per server. We're at 2 mongods per server right now (1 mongod for each shard), but frankly could probably fit a 3rd or 4th. However, I think our alternate solution will be complete soon enough we'd rather not experiment further with MongoDB.