Disqus - Latest Comments for cfeduke

Re: s3a on Spark on AWS EC2

Charles Feduke — Thu, 16 Nov 2017 18:24:00 -0000

Originally our buckets were located in us-west-1 so I had encountered a problem like this, though what I believe you are seeing is a separate problem (V2 vs. V4 authentication) and it looks to me based on the documentation that you're modifying the correct Hadoop property even though you're not dealing with Seoul or Frankfurt S3 - based on: https://hadoop.apache.org/d... I do see in later AWS docs that London is also a V4 endpoint.

In our case my configuration parameters were in spark.config which I was explicitly passing to any Spark shells or submits I ran and therefore the full key from spark.config becomes spark.hadoop.fs.s3a.endpoint - but if you were to dump your sc.hadoopConfiguration to STDOUT with your similar initialization I'd expect you to see the same key (fs.s3a.endpoint) being set to s3.eu-west-2.amazonaws.com. As a sanity check you should dump the config to a log or STDOUT just to make sure the key has the value you expect - simple things like this are what usually cost me the most time to troubleshoot because I never learn and say to myself "aww no it cannot possibly be that..."

I see in later versions of Hadoop the S3A handling has become much more robust - have a look at "Using Per-Bucket Configuration to access data round the world" https://hadoop.apache.org/d... - not sure if any of these settings exist in the version you're using, and I'll assume they don't exist in the version I was using in my article above [since this is the first time I've heard about them, and they would have been nice to have back then].

Re: s3a on Spark on AWS EC2

Charles Feduke — Thu, 16 Nov 2017 14:18:05 -0000

My first question - and this got me too - is what region is your S3 bucket in? Is it US Standard aka us-east-1? If not, you'll encounter problems without configuring S3 for the appropriate region. "Bad Request" covers a lot of possible error conditions and none of them are reported but I remember defaulting to US Standard while the bucket is actually in another region being one of them.

Re: s3a on Spark on AWS EC2

Charles Feduke — Mon, 18 Jul 2016 15:56:29 -0000

I have a deployment script that fires up a spark shell and passes the command line switch `--properties-file '/root/spark.properties'`, ends up looking something like this:

(`provided` is a directory that contains all of the JAR files that I place on all of the slaves to make my uberjar much more manageable, around 1-2MB; I use a script to copy this to the master and then run the spark-ec2/copy-dir on master to distribute to all of the slaves.)

provided=$(find my-project/provided -name "*.jar" | xargs basename | xargs printf "/root/spark/lib/%s,")
extracp=$(find my-project/provided -name "*.jar" | xargs basename | xargs printf "/root/spark/lib/%s:")

ssh -t root@$HOST "screen /root/spark/bin/spark-shell --jars '/root/$JAR_NAME,$provided' \
--total-executor-cores $CORES \
--properties-file '/root/spark.properties' \
--files '/root/$ENV.conf,/root/metrics.properties' \
--conf 'spark.driver.extraJavaOptions=-Dconfig.file=$ENV.conf' \
--conf 'spark.metrics.conf=metrics.properties' \
--conf 'spark.driver.extraClassPath=$extracp' \
--conf 'spark.driver.memory=$DRIVER_MEMORY' \
--conf 'spark.executor.extraJavaOptions=-Dconfig.file=$ENV.conf' \
--conf 'spark.executor.extraClassPath=$extracp' \
--conf 'spark.executor.memory=$EXECUTOR_MEMORY' \
--conf 'spark.executor.instances=$EXECUTORS'"

Re: Convert .iso to .img and make bootable USB

Charles Feduke — Sat, 18 Jun 2016 22:18:38 -0000

Since the Boot Camp Assistant was neutered in El Capitan [can no longer make the install USB disk, must install to internal HDD only] this is also a great way to make the Windows 10 bootable USB disk from the download ISO. Thanks for posting!

Re: Deprecating Optional.get()

Charles Feduke — Fri, 29 Apr 2016 10:25:00 -0000

I don't think it should be a common case for throwing an exception when something is missing, at least in production quality code. Rarely I can see this as being useful (perhaps in a fail fast scenario, like the actor pattern). For this sort of precondition-not-met-then-fail behavior I would use an assert.

But at least if Optional#get is used in this fashion its often easier to trace than an NPE. (Until you start putting multiple Optional#get in the same statement anyway.)

Re: Typesafe Config and maps in Scala

Charles Feduke — Thu, 28 Apr 2016 11:51:59 -0000

Have a look at Ficus: https://github.com/iheartra...

The above code becomes something like:

case class Decoder(name: String, address: String)
case class Decoders(decoders: Seq[Decoder])

Then with an import you can as[Decoders] on a ConfigObject from a Typesafe config instance and get the same thing. Magical!

Re: Deprecating Optional.get()

Charles Feduke — Thu, 28 Apr 2016 11:43:27 -0000

This is great news. I use the Scala REPL today as necessary when working with Java but dependencies can be troublesome, even with Ammonite. (I mean sometimes I use JUnit as a REPL stand-in for Java development...)

Re: Deprecating Optional.get()

Charles Feduke — Thu, 28 Apr 2016 11:01:39 -0000

Now that I think about it Optional#get should have been published with a deprecation warning right from the start. Its useful in Scala in the REPL (because: lazy!) but REPL development in Java isn't popular. I won't let Optional#get in any production code I write in Java (or Option#get in Scala) because all of the clean ways of handling the values within an Optional.

Re: s3a on Spark on AWS EC2

Charles Feduke — Thu, 21 Apr 2016 12:22:22 -0000

Seems like possibly a dependency version difference between your environment and mine, where in my environment a constructor for S3AFileSystem found in the 2.7.1 version of the JAR is used.

I've found that both AWS and Apache Hadoop developers have a terrible habit of breaking backward compatibility by changing the parameters of public methods without leaving the old methods intact.

Re: Mavericks: Day 2 with MacVim, Python, and crashing

Charles Feduke — Fri, 07 Aug 2015 11:25:07 -0000

It simply has to be. I mean its been 2 years! But I think sometime last year I went to compiling my own OSX binaries for MacVim anyway.

Re: A More Ergonomic Keyboard Strategy

Charles Feduke — Tue, 16 Jun 2015 02:51:20 -0000

I've been standing to work for a few years using a Kinesis 2 with no problems. (A typical day is over 8 hours because my favorite hobby also happens to be programming.) My arms are at 90º, wrists resting on the Kinesis 2 palm supports, but the rest of my arm is unsupported.

Re: IDictionary<TKey,TValue>, IXmlSerializable, and lambdas IDictionary에 , IXmlSerializable을, 그리고 람다

Charles Feduke — Thu, 11 Jun 2015 21:40:06 -0000

I think that zip file is lost in time and space, I'll see if its on one of my old Windows VMs. This site used to be on an AWS server until I moved it to DigitalOcean and I had forgotten that some of the older posts had links to files on the file system instead of S3.

Re: Java 8: Replace traditional for loops with IntStreams

Charles Feduke — Wed, 10 Jun 2015 10:03:46 -0000

Though I love the Java 8 stream API - I am a Scala programmer - there are a two things you should make sure your readers are aware of when replacing a standard for-loop.

- Debugging becomes more challenging.
- Parallelization is not free. Its important to note that actual performance testing must be done to determine the threshold where parallelizing a task is advantageous.

Re: Beating a Poison Ivy Reaction

Charles Feduke — Mon, 25 May 2015 12:00:20 -0000

I was actually getting ready to spray again tomorrow, since my dogs found this year's patch. I could pull it by hand; there's not a lot of it. I usually hit all of the weeds (mostly jasmine) that grows under my trees with Round Up. I haven't thought of the acidity of the soil, but mostly I don't want anything growing where the PI was growing. (Its always under these evergreen trees that start at ground level.)

Re: Whitescreen when upgrading OSX to Yosemite

Charles Feduke — Sun, 01 Feb 2015 22:47:22 -0000

I can't remember whether there was a login screen at all - I don't recall there being one. Complete the upgrade in safe mode, and it should restart afterwards and you shouldn't need safe mode any longer.

Re: Beating a Poison Ivy Reaction

Charles Feduke — Thu, 22 May 2014 10:03:30 -0000

I ordered some "Poison Ivy Fighting Suits" (just full body covers) and have two bottles of Round Up to finish what I started... once this current rash has passed.

I had forgotten about Gold Bond, I have used that in the past, as well as hydrocortisone cream. Gold Bond seems to work better than hydrocortisone for me.

Re: Moving from WordPress to Ghost

Charles Feduke — Wed, 21 May 2014 23:20:23 -0000

This article was the first hit for me when I was seeing what the path off WP to Ghost was like, and I don't think i could have found a better written post about it. Thanks!

Re: Rumors of Ruby’s Demise

Charles Feduke — Sun, 23 Feb 2014 21:39:52 -0000

In our real-world environment we solve scalability problems in Scala (streaming video, concurrency especially with Akka) and user-facing problems (website) in Ruby. I write a lot of tooling in Ruby too, because while I'm pretty productive in Scala, I'm still at least twice* as productive in Ruby.

Any rumors of Ruby nearing its end are greatly exaggerated.

(* or more, depending on how ornery scalac wants to be about the type system on any particular day)

Re: RSpeccing a JSON API gotcha

Charles Feduke — Thu, 20 Feb 2014 21:45:32 -0000

I've dug up the source code for that and updated the post with it, at the bottom of the article.

Re: Justice Beavers: ‘It’s Ironman meets Burning Man’

Charles Feduke — Thu, 16 Jan 2014 08:36:10 -0000

For local events you can check out either of the two Crossfit boxes - Rare Crossfit or Crossfit Forward (Google them); both post info on upcoming events and the athletes there are variously involved in them all. Fredericksburg has an annual mud and obstacle course run called Tribal Quest during the summer, and a series of shorter trail runs in which anyone can compete. Then there's the road races in the area that Race Timing Unlimited and other organizations schedule. When you figure in nation-wide events like Ragnar, Tough Mudder, CMC, Spartan races there's a lot of races to choose from!

Re: The Galago arrives... - Charles' Blog

Charles Feduke — Fri, 01 Nov 2013 23:18:36 -0000

I recently ordered a Gazelle Pro but the lost keyboard strikes and poor positioning of the trackpad and buttons are just too much for the price I paid. I like the configuration, and expandability (in fact had put in a 1TB WD HDD for TimeShift the day I received it), and I really want to like the machine but I cannot overcome the keyboard.

Re: Monkey Patching in Scala

Charles Feduke — Mon, 07 Oct 2013 15:41:51 -0000

C# achieves similar functionality using extension methods. I prefer the Scala approach as I've used it to good effect in my code. In Ruby its important to note that you can replace behavior in a pinch through monkey patching - which, yes, can lead to great sadness. (I have never tried overriding behavior using an implicit conversion in Scala, I suppose it would work since the compiler implicitly converts the original type at compile time.)

Re: Guard, RSpec 2, and Growl

Charles Feduke — Wed, 19 Jun 2013 17:43:16 -0000

Thanks, I've updated as appropriate.

Re: Linux has better hardware support than OS X

Charles Feduke — Mon, 17 Jun 2013 07:42:22 -0000

This is patently false. You can get *budget* hardware for less. If you configure a system to be similar specs to a MBP - including SSD, RAM (16GBs) - with a quality PC hardware provider (Lenovo, System76, Sony) they come out to about the same price, with a lower max resolution for the panel of choice and heavier.

Re: Invalid Argument

Charles Feduke — Fri, 17 May 2013 12:39:47 -0000

That looks great. I just remember watching the DAS screencast on testing bash scripts and thinking "this... is... not for me." I will probably give this a shot - I cannot always write my CLIs in Python or Ruby after all.