Disqus - Latest Comments for lukeprog

Re: Do You Feel Lucky, Punk?

lukeprog — Tue, 24 Mar 2020 18:08:27 -0000

Looks like Disqus screwed up the link I provided, but I can't edit my original comment. I meant to link here: https://goodjudgment.io/cov...

Re: Do You Feel Lucky, Punk?

lukeprog — Thu, 19 Mar 2020 16:46:09 -0000

Re: honest estimates, we now have these predictions from superforecasters. E.g. they currently think there’s a <25% chance that reported infections will rise to even 10% of US population (33M) before Apr 2021, given whatever they think the policy response will be. Of course reported cases will underestimate true cases, but the superforecasters’ forecast of US deaths is roughly consistent with their forecast of US cases (assuming a ~1% fatality rate), and their forecasts of world cases and deaths are also consistent with that.

We haven’t asked the superforecasters any conditional questions yet but I’m considering it. Anyone reading this is welcome to suggest specific (conditional or unconditional) questions we should consider posing to the superforecasters here.

Re: Replication Markets Team Seeks Journal Partners for Replication Trial

lukeprog — Tue, 08 Jan 2019 17:32:14 -0000

Is this for SCORE, or some other program?
https://www.darpa.mil/progr...

Re: Dissing Track Records

lukeprog — Fri, 20 Mar 2015 22:54:50 -0000

Google's cache seems to think "Ashley University" was recently mentioned on the CEO's LinkedIn page, even if it has since been removed:
http://imgur.com/FLLQ6Ka

Re: Dissing Track Records

lukeprog — Fri, 20 Mar 2015 16:12:04 -0000

I don't see Ashley University on his LinkedIn; has it been removed? Can you post a screenshot?

Re: Rationality: From AI to Zombies

lukeprog — Wed, 18 Mar 2015 23:12:56 -0000

There will eventually be a dead trees version, broken into 6 separate books, but it'll probably be several months before we start releasing them.

Re: Will Humans Be Able to Control Computers That Are Smarter Than Us?

lukeprog — Wed, 25 Feb 2015 13:32:49 -0000

Pretty sure I never said "nothing at this symposium is of any interest to us." IIRC, I just said "Sorry, we don't have time to attend."

But indeed, we at MIRI have found that engaging with Loosemore in particular has not been very productive. We engage with other critics instead.

Re: Building Robots With Better Morals Than Humans

lukeprog — Sat, 31 Jan 2015 14:23:06 -0000

Quick correction: Nick Bostrom isn't at MIRI, he's the director of the Future of Humanity Institute at Oxford University

Re: The Financial Times story on MIRI

lukeprog — Tue, 04 Nov 2014 13:31:16 -0000

Interesting. Now it seems like our disagreement initially sprang from the fact that you don't recognize a discontinuity between contemporary AI and AGI, whereas I do find it quite useful to distinguish the two. Have I understood you correctly? If I have, then...

"AGI" is a term that people use in a variety of ways. The operationalization of the term I happen to use these days is basically Nilsson's "employment test". To pass Nilsson's employment test, an AI program must have at least the potential (e.g. with 6 months of training, like a human) to completely automate approximately all economically important jobs.

That's still pretty vague, as definitions of future technologies must often be. (We never had a precise definition of "self-driving car," but that didn't stop us from building one.) But at least it's clear that no current AI program is an AGI by this definition — not even close. Moreover, it's clear that any AGI by this definition raises safety and security concerns not raised by any current AI program. An AGI under Nilsson's definition can do its own science, its own programming, its own marketing, its own political speechmaking.

Of course there won't be any literal discontinuity between non-AGI and AGI. The former will blur into the latter just as Homo ergaster blurred into Homo sapiens, such that it's not clear where to draw the line between them (and other human ancestors). And yet the difference between Homo ergaster and Homo sapiens is vast and important. For one thing, nuclear weapons security isn't a field needed by a planet of Homo ergasters.

All that said, of course I agree that existing AI systems are already radically changing our world and require ethical consideration and ethical engineering, it's just that existing AI systems aren't what MIRI specializes in — in part, because you and thousands of others are already devoting substantial brain power to those problems.

As for whether I'm "mining and embracing rather than distancing" myself from the contemporary AI ethics and safety engineering literature, I'll just link to some of my writings on the subject:

* Intelligence Explosion and Machine Ethics
* Transparency in Safety-Critical Systems
* Michael Fisher on verifying autonomous systems
* Paulo Tabuada on program synthesis for cyber-physical systems
* Diana Spears on the safety of adaptive agents
* Anil Nerode on hybrid systems control
* Armando Tacchella on safety in future AI systems
* André Platzer on verifying cyber-physical systems
* Greg Morrisett on secure and reliable systems
* Kathleen Fisher on high-assurance systems

All this despite the fact that MIRI doesn't at all specialize in safety and security for extant or near-future systems — again, you and many others have already specialized in those areas. MIRI specializes on very early work investigating the kinds of problems that don't arise until we've got really advanced autonomous systems, e.g. the kind capable of sufficiently general reasoning to realize that the shutdown button on its side isn't helpful for achieving its programmed goals (in expectation) and thus should be disabled. Hence our paper on corrigibility in advanced AI systems, which uses a simple toy model to open investigation into the problem, much as Butler Lampson's 1973 paper on the confinement problem did — two decades before the problem actually manifested itself "in the wild."

Re: The Financial Times story on MIRI

lukeprog — Mon, 03 Nov 2014 13:23:50 -0000

Thanks for responding and clarifying. Let me try to understand...

Certainly I don't think that the only people who are improving our odds of surviving an intelligence explosion are those fractions of people I list above. For example, not even Nick Bostrom is "explicitly doing *technical* research on the problem of how to ensure that a smarter-than-human AI has a positive impact even as it radically improves itself," and I sure as heck think Nick Bostrom is improving our odds of surviving an intelligence explosion. The same is probably true for many people working in computer security, formal methods, safety-critical systems, hybrid systems control, and other fields. I'm just not aware of any of them "explicitly doing technical research on the problem of how to ensure that a smarter-than-human AI has a positive impact even as it radically improves itself." Are you? If so, I'd very much like to read those papers.

I've read a *lot* of AI ethics literature, and I cite a good chunk of it in "Intelligence Explosion and Machine Ethics." Can you point me to a paper in the AI ethics literature that explicitly does technical research (novel non-trivial math or code) on the problem of how to ensure that a smarter-than-human AI has a positive impact even as it radically improves itself?

Or, can you point me to a paper that does that from the world of software systems engineering? I've read some of that literature (e.g. Leveson), but I'm less familiar with it than I am with the AI ethics literature.

We may be talking past each other, so it'd be helpful to me if you could point to some specific examples that you think falsify the claim I've made. I'd prefer to not go on making a false claim, but I'm just not aware of counterexamples.

Re: The 2014 Effective Altruism Summit—who stood out

lukeprog — Thu, 04 Sep 2014 17:48:11 -0000

Responding to:

> Eliezer [and Luke] do a very good job of arguing that AI safety is important and neglected, but often seem to overlook tractability, as well as the more specific issue of whether MIRI’s strategy specifically is a good one.

This is pretty fair. The problem from our perspective is that it's hard enough to say anything persuasive about astronomical stakes + x-risk landscape + AI priority in the kind of space we usually have to communicate. Arguing in favor of the type of FAI research MIRI is doing, over other FAI-targeted efforts, requires even more background than all that, and requires lots of detailed arguments about narrow facts about e.g. current AI progress, how the computer science profession works, which organizations and people are working on which exact projects right now, etc.

So if somebody has read *Superintelligence* twice and we can both afford to have a 3 hour conversation about all those details, I might be able to say something persuasive about doing FAI work vs. other efforts, but that's awfully demanding. Moreover, I think it's hard to tell which efforts toward FAI are most needed right now, so I'm not actually all that motivated to argue in favor of MIRI's FAI program *over* (say) FHI's research on AGI strategy — instead, I'd argue both efforts need to be pursued simultaneously.

In this kind of situation the usual solution is to ask the experts what looks to them like the most promising interventions right now. That's what the government does when scientists say there's some risk of major asteroid catastrophe and doing X and Y can reduce that risk.

If "the experts" in this case means the people who look generally smart and deeply familiar with the relevant considerations and work on the issue full-time rather than as a hobby, then I suppose that means the folks at FHI and MIRI and a few others. But how much should a relative outsider trust those experts? Nobody, whether at FHI or MIRI or elsewhere, has the "reliably superior performance on representative tasks" kind of expertise for AGI forecasting that plumbers and grandmasters have for plumbing and chess. Folks at FHI and MIRI mostly have "scholastic expertise" on the issue. That's more than most others have, including almost everyone in mainstream AI, but how much should a relative outsider trust those experts' judgments about what's good to do now about AGI safety? Probably less than they trust a chess grandmaster on chess, but more than they trust a machine learning expert on AGI forecasting, and I think a fairly wide range of trust vs. skepticism can be justified for different thinkers with different degrees of familiarity with the arguments, evidence, and experts.

Re: MIRI’s Experience with Google Adwords

lukeprog — Mon, 18 Aug 2014 00:06:29 -0000

Yeah, that's great!

Re: Why I may not donate to MIRI/CFAR in the future

lukeprog — Sun, 06 Jul 2014 21:43:25 -0000

> "becoming isolated from the academic mainstream"

Just a comment on this one line. I'm pretty sure things are moving in the opposite direction. The academic mainstream becoming more interested in the challenges of AGI safety due to Bostrom's book and Stuart Russell's outreach. Also, MIRI is now starting to publish papers on its technical research agenda in the major conferences and journals, e.g. see our recent AAAI paper and my forthcoming article (w/ Bill Hibbard, due in Sept 2014) in the most-read computer science magazine, Communications of the ACM.

Re: The LessWrong/MIRI community’s problem with experts and crackpots

lukeprog — Tue, 01 Jul 2014 18:59:05 -0000

Eh? That wasn't a threat. I don't have any specific doors in mind, and I'm not planning to close any doors on Chris.

Re: The LessWrong/MIRI community’s problem with experts and crackpots

lukeprog — Tue, 01 Jul 2014 18:00:16 -0000

Chris, I'd be careful throwing around the word "crackpot." Not only is it easy to misinterpret what someone else is claiming or what their reasons for claiming it are, but even if someone very clearly believes in (e.g.) psy-powers for the usual dumb reasons, calling them a crackpot in public is a risky move. You never know who you'll offend with that kind of language, and which doors you may have now closed that in the future you'll regret having closed. I've suffered for this mistake before, as have others at MIRI, as have various people in the atheist/skeptic community that you've complained about on this blog.

I don't follow nutrition closely enough to comment on Eliezer statement. As for Louie, he clearly endorses vast swaths of mainstream medicine and medical practice, and spends far more time than most people using that mainstream medical knowledge and practice to improve his own health, and if he said something about doctors being a fake profession it was probably hyperbole meant to communicate some much more narrow point that is much less crackpot-sounding than what a phrase like that can sound like in isolation. Indeed, I think the comment you're referring to was at a MIRI event sponsored by One Medical, which we decided to go ahead with because Louie went to the mainstream doctors there and liked their services quite a lot, so he could honestly recommend their services to others.

I'd recommend toning down the post for the sake of your future self, though of course I'm motivated to make that suggestion due to a concern for Eliezer's and Louie's reputations as well.

Re: Trustworthy Telepresence

lukeprog — Mon, 30 Jun 2014 18:55:27 -0000

One major problem with telecommuting is that American ISPs are too poor to get consistently good connection quality. I wonder if telecommuting is more common where ISPs are better, e.g. in Sweden or Korea or in the towns that got Google fiber. Maybe in 10 years there will be enough instances of sudden jumps in ISP quality for particular areas and this natural experiment will test the hypothesis that it's largely about connection quality.

Really, though, for myself at least I think better connection would be necessary but not sufficient.

Re: Forager Mating Returns

lukeprog — Sun, 08 Jun 2014 16:53:30 -0000

I would also like to hear recommendations for good ethnographies.

Re: Robot Econ in AER

lukeprog — Sun, 18 May 2014 15:13:12 -0000

PDF:
http://goo.gl/e2lsQy

Re: State of EA organizations

lukeprog — Mon, 28 Apr 2014 17:50:53 -0000

Which personnel are shared by MIRI and CFAR? We occasionally consult for each other, and some people are volunteers for both organizations, but I don't think we're sharing any personnel currently.

Re: Will MacAskill on normative uncertainty

lukeprog — Sun, 13 Apr 2014 12:01:18 -0000

Roughly, yes. See Wikipedia or the thesis for details.

Re: Erik DeBenedictis on supercomputing

lukeprog — Tue, 08 Apr 2014 22:54:56 -0000

Some text was lost in moving the interview to the blog, and I've now put it back. Thanks for the catch!

Re: 2013 in Review: Fundraising

lukeprog — Sun, 06 Apr 2014 17:07:38 -0000

I'll ask Malo how hard that would be to generate; stay tuned!

Re: Giving now vs. later: a summary

lukeprog — Fri, 04 Apr 2014 14:20:17 -0000

The most important reason missing from 'Reasons to give now' is "Giving to particular organizations can accelerate our learning about which causes are best to support." We'll probably get better information as time passes, but we'll get a lot more of it if we fund ongoing projects to figure out what's best to give to.

Re: Prefer Contrarian Questions

lukeprog — Wed, 05 Mar 2014 20:29:54 -0000

When you talk to people, do they not have their own opinions about the social impact of emulations?

Re: Prefer Contrarian Questions

lukeprog — Wed, 05 Mar 2014 18:00:09 -0000

Cross-posted from my Facebook wall...

I think it's legitimate to worry that MIRI-FHI folk have some contrarian views about long-term AI outcomes.

On the other hand, talking to experts in AI and forecasting has confirmed to me that approximately none of them have developed their own modular analyses (re: long-term AI outcomes) and iterated them through many rounds of criticism and fact-checking, like MIRI-FHI folk have.

So do we really have contrarian views, or do we just have unusually well-developed views on issues about which everyone else is just "making stuff up" on the spot, with no modular analysis or fact-checking?

In other words, do we have contrarian views, or did we simply ask contrarian *questions*?
[link to Robin's post above]