Disqus - Latest Comments for shaberman

Re: Avoid Using External Files in Tests

shaberman — Tue, 04 Jul 2023 14:26:58 -0000

Hi Jay; I only just now noticed your comment, sorry about that! Not sure if I'm getting notifications for these comments or not... :-/ But thanks for the note, I just fixed the link.

Re: First Principles

shaberman — Sun, 30 Aug 2020 19:09:32 -0000

"Where it feels right." :-)

Less tongue-in-cheek, I think the 80/20 principle is probably good, i.e. shoot for ~80% "perfect SOLID" and you'll get the most benefit, without being overly pedantic about the last 20% that returns less marginal utility.

Re: Avoid In-Memory Databases for Tests

shaberman — Sun, 22 Mar 2020 11:33:47 -0000

I agree in practice but I'm hesitant to give up on the theory/goal, only because it'd be really great if this did work out.

I'm not a .NET/EF developer, but I've admired LINQ from afar, because it seemed like the best chance to achieve exactly this sort of database-/memory-agnostic codebase/test suite vision. So it sucks to hear that it's not working out.

You point around transactions makes a lot of sense, although to me that's more of an implementation detail that "of course" the in-memory provider should emulate transactions (i.e. hook into whatever ER Core transaction lifecycle methods that regular db providers do), i.e. do its own mini-MVCC/whatever internally, to keep committed data separate from WIP/uncommitted data.

Because, yeah, totally agreed the behavior between the in-memory provider and a db-provider (for a validated set/subset of trusted (LINQ) operations and not $random sql) need to 100% the same, otherwise trust is lost and it just doesn't work.

I also get that some test cases exercise production code that just have to use native SQL, so those cases can't be covered in-memory anyway; that makes sense, although I'd assert that hopefully those are ~20% or less of total test cases, so if you start out with a test case using the in-memory provider, have that work for 80% of your tests, and then when you realize "ah right this code path touches native SQL" (or a "non-validated" LINQ method/operation that the in-memory provider just throws "sorry this is too complex for us to impl, use your db for this test", i.e. it should be fair for the in-memory provider to just not try to implement every esoteric query/method/etc), just move that case over to the "uses the real db" test directory, that seems like an acceptable compromise to me.

I also sympathize that, re parsers/lexers/etc, yes, building this in-memory-provider is like building a whole new db (given it has to partition committed/uncommitted data, parse non-trivial queries, etc.), and is a lot of work, which is probably why it hasn't been done before, although I'd generally assumed once a "popular enough" / "big enough project", ala .NET's EF, came along, there would be enough leverage from the vast community using the in-memory-provider, that the ROI would make sense for the centralized team to buckle down and write/maintain it.

But, I guess reality is saying that is not the case. :-)

Re: Toyota Katas Notes

shaberman — Sat, 03 Aug 2019 16:18:13 -0000

Yep, agreed. Two thoughts:

1) I'm also rereading Flow (The Principles of Product Development Flow), which is not really "port TPS to software" but "here is why [with lots of ~somewhat dry theroetical reasoning] TPS doesn't apply to software and what to do instead". I'll do a similar notes blog post if anything for my own synthesis and memory. But, yep, you've already hit on several of the key aspects (get rid of queues, short/fast iteration, less design-in-progress, etc.)

2) I'm also wondering about just removing queues all together (if/when possible). Like within engineering, we often have "the mobile engineer queue", "the frontend engineer queue", "the backend engineer queue", etc, each of which take turns being bottlenecks depending on the project/task/etc.

At my current gig, I'm on a "TypeScript everywhere!" kick (although I can't win the React Native battle yet :-)) primarily b/c I think having a common language enables breaking down some of these artificially-introduced queues and letting generalist engineers more freely flow to where the current work needs to be done (albeit probably still facilitated by platform specialists/owners who know the nitty-gritty details of each platform, like maybe an 80/20 ratio of generalists/fullstack to specialists).

Re: Continuous Queries on Dynamic Tables

shaberman — Sun, 30 Jun 2019 19:06:04 -0000

FWIW would be great to have an update at the top of the article / inline somewhere that clarifies that result-updating continuous queries are now supported (AFAIU they are).

Granted, updating point-in-time blog posts is kind of annoying, but this post is still a high search result for "flink continuous queries", so having it mention out-of-date information can be confusing.

Re: Why I’m a Static Typing Bigot

shaberman — Thu, 27 Sep 2018 16:06:13 -0000

LA LA LA LA. :-)

Re: Mono vs. Project Repos? Hurray, software sucks

shaberman — Mon, 16 Jul 2018 00:04:17 -0000

Yep, thanks! Pushed out a fix.

Re: I Forgot How Much I Don't Like Pull Requests

shaberman — Fri, 29 Jun 2018 10:38:39 -0000

Hm...the append-only nature of PRs still probably applies? E.g. in theory if someone says "fix this" on commit 1, I want to be able to apply that fix to commit 1 locally (via an interactive rebase/squash), and then repush my string of commits.

And, ideally, if the PR goes from "C1 -> C2 -> C3" to "C1' -> C2 -> C3", e.g. C1' is my fixed commit, then the comments / feedback / etc. on C2/C3 would still stay there.

However, their git hashes have changed, so I believe GitHub will just drop them? That was it's behavior awhile ago, I should try it again. But in theory it would have to, because it only has the git hash/physical id--Gerrit/other tools add a logical identifier to each commit (e.g. in the commit message or a git note).

That said, this is still better, and I'll try it as my default workflow for awhile.

Re: I Forgot How Much I Don't Like Pull Requests

shaberman — Fri, 29 Jun 2018 10:28:41 -0000

Holy shit. I knew about the "Commits" tab, which is a way to go through them ~tediously. But I was blind to this drop down...and not only that, but the drop down has hot keys! c to open it, n/p to through commit by commit. Super interesting. I will try this for awhile and see how it goes. Thank you!

Re: Tools for Remote Software Development and Pair Programming - The Zapier Engineering Blog - Zapier

shaberman — Wed, 30 May 2018 11:33:25 -0000

Shameless plug, but if you find yourself in the "mad scientist" scenario again, mirror is basically a real-time rsync that uses file system events + two-way HTTP/2 streaming to sync quickly:

https://github.com/stephenh...

It's technically targeted as a single-developer workflow of "type in IDE on laptop, build via CLI on desktop", but is effectively what you had with cron+rsync. Granted, it took quite a bit longer to write than a few lines of cron+shell. :-)

Re: Are Classes and Prototypes that Different?

shaberman — Fri, 30 Mar 2018 12:13:36 -0000

Huh, nice catch. StackExchange says technically there is a difference:

https://english.stackexchan...

But I was using "prototypical" as a synonym for "prototypal", so no, I didn't mean to insinuate a difference.

Now I can't decide if I want to make myself get used to saying "prototypal" in my head, or just copy edit all the "prototypical"s to be "prototype-based". :-)

Re: If you cherry pick, your branch model is wrong

shaberman — Mon, 19 Mar 2018 20:22:32 -0000

True, good points; I've only used gerrit for a few open source patches, but I like it.

Re: Are Classes and Prototypes that Different?

shaberman — Mon, 19 Mar 2018 12:50:50 -0000

Hehe, thanks for the nudge. We'll see, admittedly at this point it would just be a collection of gists of "wow, look at that" rather than anything too insightful, but I guess that might still be useful. I'll keep collecting gists and see how it goes.

Re: If you cherry pick, your branch model is wrong

shaberman — Thu, 01 Mar 2018 16:18:05 -0000

> In the real world you often don't

I do work in the real world :-), but, yes, I can see where projects might have multiple release branches (although it doesn't sound like fun; I knew of a healthcare system where each hospital ended up with essentially a unique codebase, and maintenance/upgrades were a nightmare. But it did generate billable hours.)

So, yes, my experience is mostly SaaS/webapps where having N releases floating around is rare. YMMV.

> use merge if you have a common merge base

Having not worked on one of these, I haven't tried it in practice, but merge-base does take N commits (e.g. git merge-base releaseA releaseB releaseC), and naively the "most recent common ancestor" commit does actually sound like a reasonable place to start a bug fix that you know will have to be merged into N releases.

Granted, that assumes you know ahead of time which releases are affected. If you don't, then, yes, cherry-pick is a pragmatic way of getting it there, vs. re-creating your branch off the newly-found most recent common ancestor.

> the issue tracker will happily tell you which changes were made on why branches

If you have that tooling, that is great, but again our experiences are differing here :-), as even with projects that have had rudimentary "issue 123 shows that SCM commit ABC fixed it", the issue trackers I've seen/used have been oblivious as to what branch/release that is on (and also unaware as to what dev/prod environments the ticket is deployed in).

...well, now I don't remember for sure, it may have recorded the branch it saw the fix on when the fix was initially pushed (and triggered the SCM -> bug tracker post-commit hook), but it definitely did not track in through subsequent merges/releases. So, the issue tracker would be a hint but not an authoritative answer.

That would be nice to have though.

Re: If you cherry pick, your branch model is wrong

shaberman — Thu, 08 Feb 2018 10:34:18 -0000

Hi Filip; agreed, I think that is a good articulation. Thanks!

Re: Java 9: Create Lists, Maps and Sets easier than ever using the new collection factory methods

shaberman — Wed, 07 Feb 2018 10:21:57 -0000

Well; shoot. What's the point of returning a mutable type when the collection is actually immutable? Seems like a mistake. Set is mutable, so Set.of should return a Set...that is mutable...

I love immutability, but we're basically lying to the type system.

Huh; the JDK designers are smart people, so surely they thought of this (to me obvious/big) con...wonder why they still went this way...

Thanks for the post!

Re: The One True Way Of Indentation

shaberman — Wed, 31 Jan 2018 10:14:15 -0000

Hey; sure, here's what I typically use:

https://github.com/stephenh...

I really want to like the google-java-format project, because I like their cross-IDE/CLI approach, and to a certain extent like their "no config flags" idealism...but IIRC they use 4-space indent, and also get way more indent happy than I would generally prefer (so code that would be more compact is, with their formatter, more strewn out).

So, for me, the Eclipse formatter (with the above settings) is still what I prefer.

Re: Micro Service Testing at Scale

shaberman — Wed, 24 Jan 2018 11:18:42 -0000

Hey Alex; agreed cloud environments make per-developer/on-the-fly environments an interesting possibility; definitely much easier to do than pre-cloud.

My hesitation would be similar to that of the "spin up all instances locally on my machine/CI box"; that it's pretty doable for N < 10 services, but once you hit a tangled web of cross-service API calls than hits ~N > 20 services, spinning up new AWS-/cloud-level services is going to be time-consuming, expensive, and likely flaky.

One idea to avoid creating brand new services each time is to have a pool of pre-spun-up environments, and then loan them out to developers/test runs as needed. E.g. if you had ~100 engineers, you might only need ~5 environments that each eng/CI box takes turns using. To actually use this for automated tested, vs. just manual sanity checking, I'd still want a way to flush/reset all of the data across all services (and programatically load in a new/minimal/test-specific data set) each time the environment is loaned out.

(Other readers of my blog might recognize this as "colored QA" environments that we used in a past system; e.g. purple QA, blue QA, etc.; it's been long enough that I forget the colors/uses of each QA environment; I also think our data setup was limited to "brand new db" or "copy of production", although we also had an extremely smaller/simpler system, so doing copy-on-writes of the production db (at the Linux LVM layer) was possible.)

Speaking of copy-on-write, I have also heard this proposed as an ideal solution to this in-the-large/shared integration environment problem: it'd work just like your proposed setup, e.g. each dev asks for their own "my-personal-stack", but if you magically assume that every production database has the ability make a near-instant/copy-on-write snapshot of it's existing data, then in theory this is quicker to setup, and you get real data.

The large/obvious complication there is that very few production databases (basically none?) natively support copy-on-write snapshots. Okay, I guess AWS's RDS does...but even that AFAIU is done at a layer "above" the database, and so really is a traditional/expensive copy, and not true copy-on-write.

The other complication with "just copy-on-write/snapshot each production database" is that I assert that'd still be a nightmare for automated tests; the data set would be constantly changing, and very complicated, so you risk tests that are hard to maintain and reason about.

Anyway, interesting to think about. :-) Thanks for the feedback!

Re: If you cherry pick, your branch model is wrong

shaberman — Tue, 16 Jan 2018 15:47:51 -0000

Sure, Jesse, no problem, feel free to reference/print out/etc. Glad you found it useful!

Re: If you cherry pick, your branch model is wrong

shaberman — Sat, 04 Nov 2017 10:39:48 -0000

Hehe, fair enough; I fixed two that I found. If I missed any others, I'd accept a pull request:

https://github.com/stephenh...

:-)

Re: Code Generation Is Not Evil

shaberman — Sat, 04 Nov 2017 10:25:36 -0000

Hi Lii; thanks for the comment.

I've not worked on sourcegen recently; couple of reasons, but primarily that I personally am writing less and less code these days (more running/leading projects/etc.) and I suck at publicizing/building a community around my projects such that others would step in and use/maintain/grow them.

That said, I'm still of the biased :-) opinion that, 7 years later, sourcegen is still the cleanest way to do Java code generation. The square team came out with something similar ~3-4 years ago (they went through a phase of writing a lot of annotation processors), but I think sourcegen still best hits the sweet spot between API-/DSL-driven (for classes/methods/fields) and string-driven (for internal method bodies) approaches.

I've not personally used EMF before, but have heard of it/am generally aware of what it does. Generating code to do data binding/etc. makes a lot of sense (e.g. that is part of what dtonator, which also used sourcegen, was doing for Tessell).

So, if you're eager to refactor their approach, I'd definitely encourage you to do a spike with the joist-util jar. Unfortunately I'm not cool enough to have it on maven central :-/ but it's in the repo.joist.ws maven repo.

If you have some initial success, I'd love to see sourcegen extracted from joist-util and promoted to it's own first-class project, e.g. cleaned up/renamed as org.sourcegen.whatever.

Re: If you cherry pick, your branch model is wrong

shaberman — Fri, 31 Mar 2017 20:02:53 -0000

A late reply, but I think rebasing for a local, unpublished branch (or just WIP commits on master if you're not a specific branch) makes a lot of sense, because yes it avoids lots of noisy/useless merges every time you "git pull".

That said, if you have a published branch, that is typically a distinct feature (e.g. feature_a), then using a merge commit to merge it into master (vs. a rebase+fast-forward merge) makes more sense IMO because it better denotes what actually happened: feature a was a distinct set of commits for awhile, and then we merged them into master.

I've worked on codebases that do both (either rebase+ff or merge) and generally prefer merging because I like seeing the fact that this was a feature branch stay in the DAG. But rebase+ff is fine as well.

Re: Writing Emails Asking for Help

shaberman — Thu, 09 Mar 2017 10:21:21 -0000

Hey Matt. Wow, I agree, your github issue really is pretty much exactly this approach/template verbatim.

> Maybe schools need to teach the "Five Paragraph Support Ticket Essay" as part of the modern STEM curriculum?

Agreed. I actually really enjoyed the Technical Communication course that I took at UNO as part of the CS curriculum. For me, it was the first "English" class that was "apply logic and rules to make your writing effective". My reaction, without realizing at the time, was basically "...wait a sec, this is like writing code, I can do this!" :-)

Re: Stuck on a Dead Framework

shaberman — Wed, 04 Jan 2017 11:05:54 -0000

Hi Hogo,

Really sorry about the delayed response; I'm not sure why I didn't see your comment sooner.

See my reply to Jasper, but I wasn't really thinking of GWT while writing this, although yes it does apply.

I agree with you that, unfortunately, while me + my team were extremely productive on our GWT + Tessell-based stack, it's very challenging to sell that to front-end JS developers as a viable, "fun" stack. I'd like to think that once they work in it awhile, they'll realize how nice it can be, but that's hard to do if, especially in today's hiring environment, they run away at the first mention of the word "GWT".

Personally, I am still bullish about GWT, and specifically the new j2cl compiler that Google is working on. I think that coupled with a Tessell-style framework that heavily stubs the UI so you can have really amazing unit tests would be very hard to beat in terms of raw developer productivity. I also think Daniel Kurka's Singular approach is enticing, and if you could have essentially a Java-/GWT-version of "React Native", it'd be extremely compelling for organizations to let their UI engineers be productive on all platforms, instead of having to hire/manager teams of specialists for each web/android/ios/etc.

That said, Typescript has, AFAICT, won the "if you have to write JS, this is the typed language to use". And it has enough "but it's still kinda/sorta JS" that most JS devs won't turn and run away like they do from GWT.

So, from a hiring perspective + technology stack perspective, I think that is a good choice. Microsoft is also very good at the developer tools game, so it's a good horse to bet on.

But I also still like GWT. If I were starting a new project and it was my choice, yes, I'd pick GWT/j2cl (fingers crossed it'll be released soon) + a rebooted-/j2cl-only Tessell, and would not have any qualms about it.

Re: Stuck on a Dead Framework

shaberman — Wed, 04 Jan 2017 10:52:12 -0000

Hi Jasper,

My post wasn't actually motivated by GWT or a GWT-based codebase, it was just a regular JS codebase that has some long-in-the-tooth frameworks (by JS standards) floating around.

That said, yes, GWT is just like any other technology and has the same lifecycle and "bet your codebase on it" pros and cons.

I do think what has hurt GWT historically is that people treated the Java -> JS transpiler and widget-based UI framework as the same thing, and so using GWT-the-transpiler meant you were defacto using the widget UI framework, which, IMO, was the biggest source of pain for people vs. the transpiler itself.

E.g. the widget framework lifecycle wound down a long time ago, but the transpiler itself is still useful IMO. (Decoupling these two aspects of transpiler vs. UI framework is exactly what their new j2cl compiler approach is doing.)

I haven't done a project in Typescript, but I like it. I have a Typescript/GWT pros/cons post formulating in my head, but haven't gotten around to typing it out yet.