DISQUS

"It also has a nutrition plan which needs to be followed or adapted to ensure you’re eating properly and eating enough."

Well, I'm screwed.

"the 4 folks who do the workouts in the background with Tony Horton range in athleticism and fitness; you pick the one you want to keep pace/form with."

Cool, I want the one with the bad back, feverish kids and %body fat like mine (my body fat at this point, is somewhat similar in makeup to a slice of cheesecake).

Seriously, I'm like you Daniel. There's this "small sample size" thing going on here where, while I trust Hoff as a reliable source, I need someone outside of the twitter clique to talk objectively about the program.

Related: I started trying to quantify calorie intake with the Livestrong iPhone program. But for some reason it was telling me that I could consume a max of 980 calories a day in order to hit my weight goals. One of us is too aggressive there.

"It also has a nutrition plan which needs to be followed or adapted to ensure you’re eating properly and eating enough."

Well, I'm screwed.

"the 4 folks who do the workouts in the background with Tony Horton range in athleticism and fitness; you pick the one you want to keep pace/form with."

Cool, I want the one with the bad back, feverish kids and %body fat like mine (my body fat at this point, is somewhat similar in makeup to a slice of cheesecake).

Seriously, I'm like you Daniel. There's this "small sample size" thing going on here where, while I trust Hoff as a reliable source, I need someone outside of the twitter clique to talk objectively about the program.

Related: I started trying to quantify calorie intake with the Livestrong iPhone program. But for some reason it was telling me that I could consume a max of 980 calories a day in order to hit my weight goals. One of us is too aggressive there.

I'm only using it to log into a small Magnolia bookmark group that stores and shares Risk Management articles.

I can see using it more in the future, but only for the dozens of non-sensitive sites that make me login (fantasy baseball, photo sharing and the like) and not something where the loss to me might be greater (paypal, online banking).

One of the issues I have with it is that I can't think of an OpenID provider that has done a good job of assuring me as to their ability to reduce the risk to me. The selling point I've seen from providers is convenience, not security. OpenID providers need to assure me of both.

I'm only using it to log into a small Magnolia bookmark group that stores and shares Risk Management articles.

I can see using it more in the future, but only for the dozens of non-sensitive sites that make me login (fantasy baseball, photo sharing and the like) and not something where the loss to me might be greater (paypal, online banking).

One of the issues I have with it is that I can't think of an OpenID provider that has done a good job of assuring me as to their ability to reduce the risk to me. The selling point I've seen from providers is convenience, not security. OpenID providers need to assure me of both.

Today's post on overcoming bias:

http://www.overcomingbias.com/2008/02/believing-too-l.html

"There are two mistakes you can make when you read a scientific paper: You can believe it (a) too much or (b) too little."

Today's post on overcoming bias:

http://www.overcomingbias.com/2008/02/believing-too-l.html

"There are two mistakes you can make when you read a scientific paper: You can believe it (a) too much or (b) too little."

The unobserved past, future, God, and Information Risk?

Belief statements could be said to be probability statements. So what you're doing here is not much different than what's happening in IRM - there is an absolutely unknown state for which we have some level of evidence, and some level of uncertainty surrounding the conclusions we can draw from that evidence. Asserting that you can move from State of Nature to State of Knowledge to State of Wisdom about God or Non-God requires that you express your certainty and/or the reasons for your lack of uncertainty along the way with each prior and posterior.

It has been my experience that most Atheists and Theists ignore the uncertainty in their evidential claims, but the Agnostic uses that uncertainty to conclude that they can draw no conclusion. But that's not an absolute. There are Atheists and Theists who are comfortable acknowledging uncertainty within the context of their belief statement, to be sure. Similarly, there are Agnostics who are express absolute certainty concerning the quality of data. The important requirement for any discussion is that all 3 states carry the burden of their uncertainty.

So while much of what you say above concerning uncertainty, beliefs and currently accepted scientific theories is not uncommon observational experience, it's not absolute. Atheists & Agnostics can be (and are) "pretentious enough" to claim State of Wisdom (i.e. they think their belief statement is most probably right with an uncharacteristically high level of certainty).

The unobserved past, future, God, and Information Risk?

Belief statements could be said to be probability statements. So what you're doing here is not much different than what's happening in IRM - there is an absolutely unknown state for which we have some level of evidence, and some level of uncertainty surrounding the conclusions we can draw from that evidence. Asserting that you can move from State of Nature to State of Knowledge to State of Wisdom about God or Non-God requires that you express your certainty and/or the reasons for your lack of uncertainty along the way with each prior and posterior.

It has been my experience that most Atheists and Theists ignore the uncertainty in their evidential claims, but the Agnostic uses that uncertainty to conclude that they can draw no conclusion. But that's not an absolute. There are Atheists and Theists who are comfortable acknowledging uncertainty within the context of their belief statement, to be sure. Similarly, there are Agnostics who are express absolute certainty concerning the quality of data. The important requirement for any discussion is that all 3 states carry the burden of their uncertainty.

So while much of what you say above concerning uncertainty, beliefs and currently accepted scientific theories is not uncommon observational experience, it's not absolute. Atheists & Agnostics can be (and are) "pretentious enough" to claim State of Wisdom (i.e. they think their belief statement is most probably right with an uncharacteristically high level of certainty).

"If what you’re claiming to do is science then your methods must have predictive power. Indeed, one of the key attributes of science is the ability to measure and quantify, then predict with some degree of accuracy that a change in one place will produce a result in another. “Risk assessment” in computer security is more like vigorous hand-waving than science, if you look at it from that perspective."

It seems to me that you're confusing scientific method with "science" there. Science has no predictive power on it's own in some anthropomorphic manner, it is simply a study about a body of knowledge. Scientific method is the process used to measure and analyze in the context of a model or theory in order to make a belief statement, no? Then, in turn, that belief statement can be tested for accuracy and/ or precision (the 'predictive power' you're claiming, I guess).

I've suggested before that you read 'Jaynes - Probability Theory, the Logic of Science'. I'm happy to hear you have an aptitude in traditional statistics - frankly, you'll probably be in a much better position after you read it than I will ever be - based on your capability to digest and use advanced subjects.

Bottom line - I believe if we continue to work through what we've got now, establishing the "laws" of security and risk (from some of your writing and inspired by http://www.overcomingbias.com/2008/01/beautiful...) we, in the end, will be no worse off than Paleontology, much of Astronomy, and similar disciplines. At worst, we'll continue towards something akin to the best of meteorology or economics. Which is a hell of a lot better than the witchdoctory we do now.

"Computer security isn’t doing that; practitioners who are fond of risk assessment models jump from a measurement to a conclusion"

I'm not sure I understand what you're saying here. Are you saying that scientists who use probability theory shouldn't be using their posteriors to draw conclusions? That probability theory and scientific method has no use in decision making? Or that the current methods for risk assessment methods you're familiar with suck it?

If it's the latter, then you'll hopefully understand that's why I get so upset when industry pundits write these sorts of statements. You're making a huge generalization here. We (myself and those who couple of dozen of folks who are trained and doing these things as part of their jobs) just might have something different, no? In probability theory we might say that you aren't accounting for the uncertainty you have around your observational data :)

So understand how we perceive the arrogance of the statements you've made in the past - we've vetted our approach against dual PHDs in stochastic methods from various universities. We've done our homework, our methods are in use and we are seeing the value. I'm not claiming they are perfect, it's a new approach that needs more and more vetting. But when someone with a lot of personal brand equity and small sample size error comes and craps out a big blanket statement that denigrates what we think actually has value? Yeah, we're going to be a little defensive.

"The use of statistics for risk management of problems in banking and underwriting works because you’re dealing with very large data-sets that are well-understood and quantified. Unlike in computer security, for example, there are excellent and detailed data-sets about the relative ages of drivers in automobile accidents - in fact the insurance industry has such detailed data-sets that they can correlate between owning a 2-seater vehicle and the likelihood of speed-related payout. But we can’t come close to doing that with computer security for two reasons: 1) The numbers just aren’t there. We’re left with handwaving B.S. like “80% of attacks come from the inside” - a nonsense number someone pulled out of their butt on a friday, which has achieved credibility through repetition among security practitioners.

Again, I'm not sure I understand the points you're trying to make here. Are you saying that it is impossible to account for the noise in data? Then someone needs to tell the rest of the scientific and assurance worlds they can't use noisy data either. Are you saying that traditional actuarial/statistical approaches don't work because they can't account for noise in data? Then I kind of agree. Are you saying that you've read a bunch of Taleb and you buy into everything he says? That you don't agree with Cox? Then we'll have to agree to disagree on an epistemic level, and discuss the value of analysis and modeling in more practical terms.

But you should understand that our approach to risk analysis initially came from a very, very large Insurance company. So when you make large generalizations about "why underwriting works" I do question the usefulness of your prior information. We know the beautiful chaos that is business decisions based on probability first hand - the size of the estimates, the uncertainties accounted for. My familial background is strong in Bayes Theorem (my Dad being early on in NMR) so claiming that probability theory cannot help science cope with similarly noisy data isn't going to resonate, either (sorry for the pun). If anything, I'll continue to assert that we should pursue analytic functions using the best that science and probability theory have to offer us - even if my current model turns out to not be the best one mirroring reality. We should account for our noisy data in the same way that they do.

You're a self-accliamed Feynman fan. Seek what he would do with noisy data and uncertainty. I don't think he'd as carelessly dismiss the stochastic approach as a useful tool as you have. http://www.springer.com/west/home/statistics?SG...>

To that extent we have the tools in probability theory. We know how to best use those tools thanks to scientific method. We don't all have the laws and subsequent models. Our data could be of better quality. But having industry pundits crap all over the fledgling efforts that have been made and telling us to just give up - yeah, I guess that's what history proves we should do.

"2) Security is a dynamic environment. When you’re up against an intelligent enemy, your ability to fall back on how past statistics indicate the likely future is extremely impaired. This is why military commanders seldom get away with telling the enemy, “based on past performance, we win.” Our enemy innovates. Consequently, a Mac might be “safer” today and less safe tomorrow. Meanwhile, teen-age male drivers are not likely to get dramatically worse or better in the next decade."

Paraphrasing a statement you made above: We all know that analytic functions don't predict the future, right?

The purpose of analysis is to help people make decisions (synthesis, as Kant puts it). Any probability statement is a belief statement about current state or some past/future state based on evidence. It is a belief that is hopefully tested by scientific method (implementation of Bayes and Jaynes' desiderata is said to be an analogue of scientific method). Expressed in that belief is not only the probability that it is wrong, but the uncertainty (key) surrounding the data and the concluding belief statement. If your problem is with the accuracy of current belief statements from lousy risk models - I can't agree more. If you're saying (as it appears to me you have) that it's impossible to create accuracy using scientific method and probability theory - I'm going to continue to argue with you until you or someone else has strong evidence that Bayes/Cox/Jaynes' theorems are wrong.

But even if you do end up creating a revolution in probability theory and show Cox or Bayes to be wrong, then hopefully you will arrive at a conclusion that the journey is worth the effort. I'll personally celebrate because we've arrived at something more useful.

"And you, sir, merely come off as defensive of your dogma."

Guilty to my own shame. But aren't we both? The only question is which dogma is beneficial - one that suggests we apply scientific method to models and theories until we have some level of accomplishment - or the dogma that insists upon outright rejection and suggests we just put our faith in their personal knowledge instead?

I'd rather use science than shamanism with you (or Donn Parker, or the PCI council, or whomever) as the shaman - even as much as I've been taught to respect your past work and knowledge by people we both know. And that's what pisses me off Marcus. You're smarter than this. You'd never stand for someone telling you to drop science and put your faith in them to be your priest. And frankly, if you were in my position I doubt you'd be as charitable.

"If what you’re claiming to do is science then your methods must have predictive power. Indeed, one of the key attributes of science is the ability to measure and quantify, then predict with some degree of accuracy that a change in one place will produce a result in another. “Risk assessment” in computer security is more like vigorous hand-waving than science, if you look at it from that perspective."

It seems to me that you're confusing scientific method with "science" there. Science has no predictive power on it's own in some anthropomorphic manner, it is simply a study about a body of knowledge. Scientific method is the process used to measure and analyze in the context of a model or theory in order to make a belief statement, no? Then, in turn, that belief statement can be tested for accuracy and/ or precision (the 'predictive power' you're claiming, I guess).

I've suggested before that you read 'Jaynes - Probability Theory, the Logic of Science'. I'm happy to hear you have an aptitude in traditional statistics - frankly, you'll probably be in a much better position after you read it than I will ever be - based on your capability to digest and use advanced subjects.

Bottom line - I believe if we continue to work through what we've got now, establishing the "laws" of security and risk (from some of your writing and inspired by http://www.overcomingbias.com/2008/01/beautiful...) we, in the end, will be no worse off than Paleontology, much of Astronomy, and similar disciplines. At worst, we'll continue towards something akin to the best of meteorology or economics. Which is a hell of a lot better than the witchdoctory we do now.

"Computer security isn’t doing that; practitioners who are fond of risk assessment models jump from a measurement to a conclusion"

I'm not sure I understand what you're saying here. Are you saying that scientists who use probability theory shouldn't be using their posteriors to draw conclusions? That probability theory and scientific method has no use in decision making? Or that the current methods for risk assessment methods you're familiar with suck it?

If it's the latter, then you'll hopefully understand that's why I get so upset when industry pundits write these sorts of statements. You're making a huge generalization here. We (myself and those who couple of dozen of folks who are trained and doing these things as part of their jobs) just might have something different, no? In probability theory we might say that you aren't accounting for the uncertainty you have around your observational data :)

So understand how we perceive the arrogance of the statements you've made in the past - we've vetted our approach against dual PHDs in stochastic methods from various universities. We've done our homework, our methods are in use and we are seeing the value. I'm not claiming they are perfect, it's a new approach that needs more and more vetting. But when someone with a lot of personal brand equity and small sample size error comes and craps out a big blanket statement that denigrates what we think actually has value? Yeah, we're going to be a little defensive.

"The use of statistics for risk management of problems in banking and underwriting works because you’re dealing with very large data-sets that are well-understood and quantified. Unlike in computer security, for example, there are excellent and detailed data-sets about the relative ages of drivers in automobile accidents - in fact the insurance industry has such detailed data-sets that they can correlate between owning a 2-seater vehicle and the likelihood of speed-related payout. But we can’t come close to doing that with computer security for two reasons: 1) The numbers just aren’t there. We’re left with handwaving B.S. like “80% of attacks come from the inside” - a nonsense number someone pulled out of their butt on a friday, which has achieved credibility through repetition among security practitioners.

Again, I'm not sure I understand the points you're trying to make here. Are you saying that it is impossible to account for the noise in data? Then someone needs to tell the rest of the scientific and assurance worlds they can't use noisy data either. Are you saying that traditional actuarial/statistical approaches don't work because they can't account for noise in data? Then I kind of agree. Are you saying that you've read a bunch of Taleb and you buy into everything he says? That you don't agree with Cox? Then we'll have to agree to disagree on an epistemic level, and discuss the value of analysis and modeling in more practical terms.

But you should understand that our approach to risk analysis initially came from a very, very large Insurance company. So when you make large generalizations about "why underwriting works" I do question the usefulness of your prior information. We know the beautiful chaos that is business decisions based on probability first hand - the size of the estimates, the uncertainties accounted for. My familial background is strong in Bayes Theorem (my Dad being early on in NMR) so claiming that probability theory cannot help science cope with similarly noisy data isn't going to resonate, either (sorry for the pun). If anything, I'll continue to assert that we should pursue analytic functions using the best that science and probability theory have to offer us - even if my current model turns out to not be the best one mirroring reality. We should account for our noisy data in the same way that they do.

You're a self-accliamed Feynman fan. Seek what he would do with noisy data and uncertainty. I don't think he'd as carelessly dismiss the stochastic approach as a useful tool as you have. http://www.springer.com/west/home/statistics?SG...>

To that extent we have the tools in probability theory. We know how to best use those tools thanks to scientific method. We don't all have the laws and subsequent models. Our data could be of better quality. But having industry pundits crap all over the fledgling efforts that have been made and telling us to just give up - yeah, I guess that's what history proves we should do.

"2) Security is a dynamic environment. When you’re up against an intelligent enemy, your ability to fall back on how past statistics indicate the likely future is extremely impaired. This is why military commanders seldom get away with telling the enemy, “based on past performance, we win.” Our enemy innovates. Consequently, a Mac might be “safer” today and less safe tomorrow. Meanwhile, teen-age male drivers are not likely to get dramatically worse or better in the next decade."

Paraphrasing a statement you made above: We all know that analytic functions don't predict the future, right?

The purpose of analysis is to help people make decisions (synthesis, as Kant puts it). Any probability statement is a belief statement about current state or some past/future state based on evidence. It is a belief that is hopefully tested by scientific method (implementation of Bayes and Jaynes' desiderata is said to be an analogue of scientific method). Expressed in that belief is not only the probability that it is wrong, but the uncertainty (key) surrounding the data and the concluding belief statement. If your problem is with the accuracy of current belief statements from lousy risk models - I can't agree more. If you're saying (as it appears to me you have) that it's impossible to create accuracy using scientific method and probability theory - I'm going to continue to argue with you until you or someone else has strong evidence that Bayes/Cox/Jaynes' theorems are wrong.

But even if you do end up creating a revolution in probability theory and show Cox or Bayes to be wrong, then hopefully you will arrive at a conclusion that the journey is worth the effort. I'll personally celebrate because we've arrived at something more useful.

"And you, sir, merely come off as defensive of your dogma."

Guilty to my own shame. But aren't we both? The only question is which dogma is beneficial - one that suggests we apply scientific method to models and theories until we have some level of accomplishment - or the dogma that insists upon outright rejection and suggests we just put our faith in their personal knowledge instead?

I'd rather use science than shamanism with you (or Donn Parker, or the PCI council, or whomever) as the shaman - even as much as I've been taught to respect your past work and knowledge by people we both know. And that's what pisses me off Marcus. You're smarter than this. You'd never stand for someone telling you to drop science and put your faith in them to be your priest. And frankly, if you were in my position I doubt you'd be as charitable.

@Carl,

So it would seem. My assumptions were WAY to hasty. Thank you for being understanding. RE:

Guess: an opinion that one reaches or to which one commits oneself on the basis of probability alone or in the absence of any evidence whatever.

To me, the difference in belief statement vs. guess has to do with the quality of evidence (data). Probability suggests that there is some evidence, no matter how small or useful. Guess suggests that there is no evidence at all to prove or disprove the probability statement.

Semantic, to be sure, but "guess" carries such a negative connotation as to render a probability statement as useless. IMHO, that's unfair.

@Carl,

So it would seem. My assumptions were WAY to hasty. Thank you for being understanding. RE:

Guess: an opinion that one reaches or to which one commits oneself on the basis of probability alone or in the absence of any evidence whatever.

To me, the difference in belief statement vs. guess has to do with the quality of evidence (data). Probability suggests that there is some evidence, no matter how small or useful. Guess suggests that there is no evidence at all to prove or disprove the probability statement.

Semantic, to be sure, but "guess" carries such a negative connotation as to render a probability statement as useless. IMHO, that's unfair.

Oh, and the fact that "educated guess" and "probability statement" are not synonyms and most people act like they are. "Guess" is one of those words InfoSec folks like to throw around without thinking about what it means, addressing the root cause of their uncertainty, and/or accounting for it in their data.