Disqus - Latest Comments for dreamnotoftoday

Re: Just Be Awesome

Rob Spectre — Wed, 19 Aug 2015 10:58:38 -0000

That insight was so trenchant I just spilled my oolong iced tea over my latest term sheet, Reece.

I bet that was *exactly* what you needed!

Re: Alphabet+

Rob Spectre — Tue, 11 Aug 2015 19:55:55 -0000

This insight is so trenchant I just kicked my neighbor in the face.

That'll teach him to borrow my garden weasel without asking.

Re: Announcing New Investors and Our $130 Million Series E Round

Rob Spectre — Thu, 30 Jul 2015 11:26:56 -0000

Much obliged sir. Lot of fun still ahead.

Re: Introducing DELETE, A Privacy Feature for Deleting Call and Message Activity

Rob Spectre — Tue, 16 Dec 2014 11:10:03 -0000

Good eye Burke - much obliged.

Re: Using Natural Language Processing for Better SMS Interfaces Using Twilio and Python’s TextBlob

Rob Spectre — Fri, 27 Jun 2014 13:40:17 -0000

Losing a weekend is definitely easy to do with TextBlob - feature rich with a simple interface. Hope you let us know what you ended up building if you give it a whirl this weekend, Mick.

Re: Sending SMS From Your Go app

Rob Spectre — Fri, 20 Jun 2014 09:45:06 -0000

Oh biscuits - I better up my game.

No one stays on top long in New York.

Re: The Road Coders Survival Kit

Rob Spectre — Fri, 21 Mar 2014 20:28:31 -0000

Oh *man* - I'm such a huge fan of Grid-It's. So much so, we made it one of the pieces of swag you got at the last Twiliocon. So great for all the dongles we have to carry around as presenters.

Re: The Road Coders Survival Kit

Rob Spectre — Fri, 21 Mar 2014 16:23:45 -0000

Oh man - these Tom Bihn bags look amazing.

Thanks for sharing Charlie. Will need to find a store to scope the Brain Bag in person.

Re: TwilioCon 2013: Announcing Picture Messaging, Price Drop, Business Critical Apps and More

Rob Spectre — Wed, 18 Sep 2013 18:41:44 -0000

Hey Derek!

Are you using the new /Messages resource? You can use it with client.SendMessage() instead of client.SendSmsMessage().

Can you give that a try?

Re: Billing Incident Post-Mortem: Breakdown, Analysis and Root Cause

Rob Spectre — Wed, 24 Jul 2013 19:39:47 -0000

It was serious indeed and we are dedicated to making this right for everyone who suffered from this event.

In our previous update, we promised to make whole those who were affected including refunding overdraft or service fees incurred from the incident. If this happened to you Mark, please reach out to help@twilio.com so we can make this right for you.

Re: Twilio incident and Redis - Antirez weblog

Rob Spectre — Wed, 24 Jul 2013 19:09:18 -0000

Really appreciate you writing this Salvatore - impressive commitment to your project to weigh in when something like this happens after midnight when you're supposed to be on vacation. Much obliged sir.

Think it may have been in reaction to some misconception on Twitter today about our billing system's architecture, which I can definitely understand from a cursory read of the incident report we filed last night (http://www.twilio.com/blog/.... I'm not sure how the perception that Redis was the primary store of account balance data started - suspect I could have explained it better in the incident report. My bad.

Thanks for the thorough read of the report Salvatore. Your analysis is spot on - two issues that contributed to our redis-master failure was the host machine's incapability to survive the load of the simultaneous sync of all slaves and an incorrect configuration that turned the redis-master into a slave of itself. This rendered all account balances in the in-flight stack to zero and the billing system did not check the balance with the source of truth on disk, the relational datastore serving as the primary ledger.

However, these were not the only two that created the incident. Had one of quite a few things had not happened - some that had to do with our Redis cluster, some that did not - the incident would not have occurred. A number of corrective actions are being implemented now to prevent such a dangerous failure again. It was definitely *not* a case of Twilio "storing banking information only in memory" as some have suggested. And even further, we are unequivocally *not* suggesting Redis is to blame for the billing incident last week.

*We* are to blame for the billing incident last week. Redis was one piece of software implemented in our solution that came up short for our customers and our own expectations of ourselves. Part of owning up to that responsibility is a candid and accurate description of what happened and what we intend to prevent it from happening again.

Just to be crystal clear in case any of your readers are confused:

- Redis is a part of Twilio's architecture for a double-bookkeeping billing system in addition to the primary ledger in a relational datastore.
- Redis has been great to us and we intend to keep using it with the corrective action we specified in the incident report.
- Redis does not serve as our primary account balance for Twilio customers.

Finally Salvatore - we would love for you to come by Twilio any time you are in San Francisco. We have dinner every Wednesday night - would be awesome to have you join us.

Re: Billing Incident Post-Mortem: Breakdown, Analysis and Root Cause

Rob Spectre — Wed, 24 Jul 2013 16:03:27 -0000

The full arch is a tough thing to communicate accurately in an incident report like this - not sure I did a great job here. To be clear, our relational datastore does serve as our primary source of truth post-flight (e.g. after a call or SMS message is compelted). The incident that occurred here was with our in-flight balance processing which is served by the Redis cluster which I would agree does not have the level of transaction safety appropriate to serve as the primary ledger of a billing system. "Twilio uses Redis for primary account balances" is not an accurate characterization, but I can understand how one would draw such a conclusion from a cursory read of this incident report.

Regardless, thank you very much for the feedback. Think we will be doing a lot of re-evaluation of the stack in coming months - really appreciate you taking the time to share your thoughts on the architecture.

Re: Billing Incident Post-Mortem: Breakdown, Analysis and Root Cause

Rob Spectre — Wed, 24 Jul 2013 06:40:04 -0000

Good eye - much obliged.

Re: Billing Incident Post-Mortem: Breakdown, Analysis and Root Cause

Rob Spectre — Wed, 24 Jul 2013 06:39:31 -0000

Quite a few of us worked on putting this together over the course of the weekend. Hope you don't feel it is inhuman - definitely a big group effort to get the information together.

Re: Billing Incident Update

Rob Spectre — Fri, 19 Jul 2013 09:37:32 -0000

Totally hear what you're saying Ken. Can you connect with our support team at help@twilio.com and explain the situation.

We can likely help with a configuration setting on your account to prevent such an event.

Re: Billing Incident Update

Rob Spectre — Fri, 19 Jul 2013 08:55:42 -0000

Very sorry for this John.

We'll make it right.

Re: Billing Incident Update

Rob Spectre — Fri, 19 Jul 2013 08:55:31 -0000

Refunding is already underway. If you haven't seen it post to your account yet, please shoot an email to help@twilio.com so we can confirm your refund.

Re: Billing Incident Update

Rob Spectre — Fri, 19 Jul 2013 08:54:55 -0000

Good feedback. We'll get it fixed.

Re: Billing Incident Update

Rob Spectre — Fri, 19 Jul 2013 08:54:39 -0000

I think that's a natural reaction to an incident like this. Truth is we let you down.

We are ashamed indeed - this was not our best work. We will make it right for you Doug. If you haven't already, please shoot us an email at help@twilio.com so we can get you sorted directly.

Re: Building Your First App: Designing the Skin

Rob Spectre — Thu, 21 Apr 2011 17:46:47 -0000

Sure - you can mix and match a bunch of different UI concepts into the same app.

Re: Video: Apparently, Blood Is NOT Thicker Than Baseball Allegiances

Rob Spectre — Wed, 06 Apr 2011 19:07:12 -0000

You call it cruel.

I call it responsible parenting.

Re: http://blog.radbox.me/post/2068938662

Rob Spectre — Tue, 22 Mar 2011 17:57:42 -0000

Re: api - is there a client auth mechanism in place?

Re: http://blog.radbox.me/post/2068938662

Rob Spectre — Tue, 22 Mar 2011 17:50:07 -0000

When are you guys making a Radbox Boxee app? :)

Re: Building Your First App – Release It!

Rob Spectre — Mon, 21 Mar 2011 10:56:15 -0000

Default style is available with a simple RSS app. If you don't want to customize your experience, that is always an option. As far as the obnoxiousness of editing XML, I'm don't exactly follow how HTML is less so, but to each his own I suppose.

Re: Getting Made into the Crew

Rob Spectre — Mon, 19 Apr 2010 15:40:00 -0000

Killer! Looking forward to seeing your app soon!