the responsibility we have as software engineers

I had the chance to chat this week with the very awesome Kate Heddleston who mentioned that she’s been thinking a lot about the ethics of being a software engineer, something she just spoke about at PyCon Sweden. It brought me back to a post I wrote a few years ago, where I said:

There’s this continued and surprisingly widespread delusion that technology is somehow neutral, that moral decisions are for other people to make. But that’s just not true. Lessig taught me (and a generation of other technologists) that Code is Law


In 2008, the world turned against bankers, because many profited by exploiting their expertise in a rapidly accelerating field (financial instruments) over others’ ignorance of even basic concepts (adjustable-rate mortgages). How long before we software engineers find our profession in a similar position? How long will we shield ourselves from the responsibility we have, as experts in the field much like experts in any other field, to guide others to make the best decision for them?

Well, I think that time has come.

Everyone uses software, very few people understand it. What seems obvious to a small elite group is completely opaque to the majority of the world. This gap is incredibly hard for us, the software engineering elite, to see. A few examples:

  • The Radiolab Podcast did a wonderful piece – Trust Engineers – where they explored the case of Facebook running experiments on its newsfeed. For non-engineers, there’s an incredible feeling of breached trust upon realizing that a set of flesh-and-blood humans have that much control over the algorithm that feeds them daily information. (And, for that matter, to most researchers used to interacting with an IRB, there’s complete shock at what Facebook did.) For most engineers, including a number of very good and ethical people at Facebook, it’s surprising that this is even an issue.
  • A couple of years ago, a friend of a friend – who happens to be a world-renowned physician and research scientist – asked me: “Ben, can the system administrators at work read my email? Even if they don’t have my password?” The answer is yes and yes. This is obvious to us engineers, so much so that we don’t even think twice about it. To a non-engineer, even an incredibly smart person, this is absolutely non-obvious.
  • A close friend, another very smart person, was discussing something with his young child recently, and I overheard “if you don’t know, ask the computer, the computer knows and it’s always right.” Where do I begin?

We, software engineers, have superpowers most people don’t remotely understand. The trust society places in us is growing so rapidly that the only thing that looks even remotely similar is the trust placed in doctors. Except, most people have a pretty good idea of the trust they’re placing in their doctor, while they have almost no idea that every time they install an app, enter some personal data, or share a private thought in a private electronic conversation, they’re trusting a set of software engineers who have very little in the form of ethical guidelines.

Where’s our Hippocratic Oath, our “First, Do No Harm?”

I try very hard to think about this in my own work, and I try to share this sense of duty with every engineer I mentor and interact with. Still, I don’t have a good answer to the core question. Yet it feels increasingly urgent and important for us to figure this out.

browser extensions = user freedom

The web browser has become the universal trusted client. That can be good: users can mostly rely on their browsers to isolate their banking site from the other web sites they visit. It can also be bad for users’ freedom: Facebook can encourage the world to add “Like” buttons everywhere, and suddenly users are being tracked across the web by Facebook IFRAMEs. Web browsers don’t have to send cookies to Facebook in those IFRAMEs, but if they don’t, they will appear to be broken, and so browsers tend to stick to existing standard behaviors.

Browser extensions, or add-ons, can help address this issue. They can modify the behavior of specific web sites by making the browser defend user control and privacy more aggressively: they can block ads, block flash, block cookies for certain domains, add extra links for convenience (i.e. direct links to Flickr’s original resolution), etc.. Browser extensions empower users to actively defend their freedom and privacy, to push back on the more egregious actions of certain web publishers.

Except in the mobile space. Think about the iPhone browser. Apple disallows web browsers other than Safari, and there is no way to create browser extensions for Safari mobile. When you use Safari on an iPhone, you are using a browser that behaves exactly like all other iPhone Safaris, without exception. And that means that, as web publishers discover improved ways to track you, you continue to lose privacy and control over your data as you surf the Web.

This situation is getting worse: the iPad has the same limitations as the iPhone. Technically, other browsers can be installed on Android, but for all intents and purposes, it seems the built-in browser is the dominant one. Simplified computing is the norm, with single isolated applications, never applications that can modify the behavior of other applications. Thus, no browser extensions, and only one way to surf the web.

Who needs trusted computing and remote software attestation when we’re all using curated devices that, by policy and/or convenience, run only one standard, unmodified web browser?

As computing moves to the web, the Web Browser becomes the operating system, and users need ways to tweak the operating system’s behavior, or web publishers will gain far too much power. We need browser extensions. We need them to be super easy to install, like Google Chrome extensions. We need them to have extensive power to modify browser behavior, like Firefox extensions. We need them to work on mobile platforms, now.

Browser extensions used to be fun little hacks. They’ll soon be necessary to maintain a balance of power between users and publishers on the Web.

(There’s a rumor going around that Apple is about to introduce an extension framework for Safari. Whether or not this extension framework is compatible with the iPhone/iPad will be a big deal, in my opinion.)

the genius of Steve Jobs: he makes you want the lock-in

Steve Jobs is a genius for many reasons, but one reason that may be under-appreciated is his unparalleled ability to convince users that he’s locking them into his platforms for their own good.

Consider Jobs’s latest letter explaining why he won’t accept Flash on the iPhone/iPad. Most of the letter is right on: Adobe’s Flash technology on the web is slow, not open, and best replaced by HTML5. Apple has a history of ditching old technologies and pulling industry forward: they killed the floppy disk on the iMac when everyone thought it was too early, they moved to flat screens across the line before others, they embraced USB peripherals and DVI video faster than everyone else, etc. Same thing here: Flash is bloated, slow, has serious security problems, and flies in the face of the open, copy-and-paste-the-source-code Web that we know and love. Apple is leading the way in removing it from its web browser. Good for them.

And then Jobs transitions:

Adobe also wants developers to adopt Flash to create apps that run on our mobile devices. We know from painful experience that letting a third party layer of software come between the platform and the developer ultimately results in sub-standard apps and hinders the enhancement and progress of the platform.

Except this is a sleight of hand: it’s just not true. Apple already has a mechanism to control for app quality: they can reject any app for any reason. So why add this additional level of control? Why automatically reject an app that happens to be originally built using Adobe’s cross-compiler, even if that app is good? The reason is in the next paragraph:

This becomes even worse if the third party is supplying a cross platform development tool.

Again, if an app is crappy, the user simply won’t buy it. So the problem for Apple is not that cross-platform necessarily means bad app (it doesn’t), but that cross-platform means…. cross-platform. If developers can easily create apps that run on iPhone, Android, WebOS, etc. then Apple has lost a bit of its lock-in. It’s easier for you to switch to a different platform. And that is something Jobs doesn’t want.

This is by no means the first time Jobs has deftly maneuvered to maximize user lock-in while making you think it’s for your own good. Remember DRM in iTunes? This was supposedly because of the music labels insisting on DRM. We were told that, without tight control over the spread of these music files, Jobs could never have convinced the labels to move legal music online. And we were thankful that Jobs had kicked the music industry into the 21st century, and most of us were willing to swallow the bitter pill of DRM. Except, now that iTunes sells non-DRM’ed songs, Apple has maintained all sorts of limitations that appear to be gratuitous. iTunes only syncs with iPods/iPhones, and Apple went out of its way to prevent Palm’s Pre phone from connecting to iTunes. Why? Because that would make it a little bit too easy for users to stop using Apple products.

The sad thing is, I think most of Apple’s products produce sufficient lock-in thanks to quality alone. I continue to use Apple desktop and laptops, because they’re that much better. But the artificial lock-in of the iTunes/iPod/iPhone/iPad chain is beginning to make me very uncomfortable. Jobs has convinced a lot of people that this lock-in is for their own good. I don’t believe him.

UPDATE: typo fix, thanks Hacker News.

What Nick Carr doesn’t get: hobbyists are the canary in the coal mine

I told myself I wouldn’t write about the iPad anymore, but I have to.

Nick Carr has joined the John Gruber club, by calling us Luddites:

What these folks are ranting against, or at least gnashing their teeth over, is progress – or, more precisely, progress that goes down a path they don’t approve of. They want progress to, as Bray admits, follow their own ideological bent, and when it takes a turn they don’t like they start grumbling like granddads, yearning for the days of their idealized Apple IIs, when men were men and computers were computers.


While progress may be spurred by the hobbyist, it does not share the hobbyist’s ethic.

Back in 1999, I used to pitch open-source software to big companies. I heard very similar retorts: why would customers want to hack at the code? Let professionals handle this, this is not a job for hobbyists! Over the years, I learned the most important response to that misguided opinion: it’s not about whether hobbyists can make modifications, it’s about whether someone other than the original manufacturer can.

With open-source software, it’s not that you will necessarily hack the code yourself, but the fact that you can means that you can also hire the professional you choose to do the job for you.

Similarly, if a hybrid car comes with a sealed hood that you, a hobbyist, can’t get to even with the right tools, then you also can’t choose the professional you want to service your car. Sure, you probably want to stick with the professionals “authorized” by the auto manufacturer, especially if you’re not an expert yourself. But nothing legal or technical is going to prevent you from choosing some “unapproved professional” you trust. If you can tinker with it, then you can hire someone else to do it for you. You may be taking a risk of voiding the warranty and irreversibly breaking your hybrid car, but that’s your choice and your risk to take.

And the same thing applies to the iPad. Sure, the average consumer is probably best off using apps that have been vetted by Apple. But if the technology harshly enforces this constraint, if you can’t hack at it, then you can no longer pick the professional of your choice to hack at it, and you depend entirely on Apple. That’s bad for tinkerers and hobbyists, as I described in an earlier post, and thus it’s also bad for progress because it removes choice.

So, no, Nick, we’re not saying that every device must make every effort to be hobbyist friendly. What we’re saying is that when a company goes out of its way to prevent hobbyists from tinkering, they’re also going out of their way to prevent end-users from choosing their professional provider. That’s a departure from progress as we’ve known it throughout history. And it will mean more decisions that don’t really benefit users, like the retroactive removal of apps that are “not quite useful enough,” and similar abuses that are all too common when one company maintains such airtight control. It was bad for progress when mobile phone companies controlled their phones in this way, and it’s bad for progress when Apple does it, too.

Remember, when radios started receiving only instead of also transmitting, that was the company building a simpler device, removing features to make it simpler and cheaper. The Apple iPad is not an unprogrammable computer. It’s a programmable computer with an added feature, DRM, that actively prevents you from programming it if you’re not approved by Apple. It’s more complicated to build that way, because it’s actively blocking you from trusting another company to vet your apps.

You call that progress?

Myth: the app store will protect you and prevent user confusion

An interesting thing happened with the Apple AppStore this weekend:

This weekend, as hundreds of thousands of people explored their iPads […] they found […] an application called Facebook Ultimate, featuring a sleek version of the familiar ‘f’ logo. The application quickly rose through the ranks to become one of the App Store’s top selling iPad applications. Unfortunately, it soon became apparent to these users that the application simply wasn’t very good, and that it wasn’t created by Facebook at all.


the $2.99 application rose to become the #7 top paid app on the App Store. The app received many poor reviews from upset users, plenty of which warned others that this was not the app they were looking for, but that apparently didn’t stop people from downloading it.

Tell me again how the App Store protects users?

“It’s a tradeoff” and other uni-dimensional thinking

Many folks, like John Gruber, are responding to criticisms of the iPad’s closed ecosystem with the “it’s a tradeoff” idea: to have such a great computer, you need to lock it down. Some use the argument that Linux has never conquered the desktop, so there, open is incompatible with good usability (I’m looking at you engadget).

That is some twisted backwards logic.

Apple needs to remove apps it finds “not useful enough” for the iPad to work well? Apple needs to be the sole app distributor for the iPad to be so desirable? It would make the iPad worse if, say, Firefox were allowed to compete with Safari on it? No, absolutely not. There is no inherent tradeoff. Apple chose to close the ecosystem. They could have had just as good a product with an open ecosystem… or, gasp, maybe an even better product where fixing a bug doesn’t require approval from the appstore overlords.

So enough with the uni-dimensional thinking. Those of us criticizing the iPad aren’t saying it’s all bad. If it were all bad, we wouldn’t be spending any time worrying about its impact on computing, because, if it were all bad, it wouldn’t have an impact. John Lilly and Ben Fry, who also expressed issues with the iPad, are probably getting one. I may well be getting one.

Apple is very good at bundling a little bit of badness with a lot of goodness and making you think there’s an inevitable tradeoff: iPod DRM, iPhone approved apps to prevent the phone network from being “taken down by a rogue app,” etc. But the only tradeoff here is that, if Apple opened the ecosystem, they would make a little bit less money. (Apple does not benefit as much as others do from an open ecosystem because their closed hardware is already so freaking popular.) For the user, the closed ecosystem is not a trade-off, it’s an unnecessary constraint.

The Accidental Tinkerer, Unexpected Lock-in, and Fatherhood

Ben Fry recently explained his concerns about the iPad:

I want to build software for this thing. I’m really excited about the idea of a touch-screen computing platform that’s available for general use from a known brand who has successfully marketed unfamiliar devices to a wide audience.
It represents an incredible opportunity, but I can’t get excited about it because of Apple’s attempt to control who creates for it, and what they can create for it. Their policy of being the sole distributor of applications, and even worse, requiring approval on all applications, is insulting to developers.
I find it offensive on a very basic level, because I know that if such restrictions were in place when I was first learning to write software — mostly on Apple machines, no less — I would not have a career in the field.

John Lilly followed up brilliantly:

In a nutshell, what worries me about the trajectory of computing is not so much the emergence of tightly-controlled, non-tinkerable boxes, but the presumption that “normal people” don’t ever want to tinker, don’t want to be bothered with understanding how things work. I think it’s not true, really — certainly not for everyone — but I even think that this distinction between “normal people” and “tinkerers” or “techies” or “makers” is bogus at best, and really dangerously corrosive at worst.
It’s not like I was born an engineer — the instinct to fiddle with things isn’t something we’re born with. I became a tinkerer because I was exposed to surfaces that allowed — that invited — it. I figured out that I liked tweaking and building and creating because I got a bunch of chances to do that stuff, from hardware to software and everything in between. I knew I could do it because Dad modeled that behavior, but also because the stuff we had around the house was inspectable and malleable.
We all have the potential inside us to make things. But we’re not born into the world as makers — the world around us — the people in it and the artifacts in it — help us to discover what we can be.

I don’t know that I agree 100% with John: not everyone is a tinkerer. But, for sure, we need “surfaces that invite tinkering,” otherwise those who would be tinkerers might never discover it.

I was a tinkerer from an early age, but most of my tinkering in the physical world sucked, because, well, I don’t have good instincts about physics or analog things: I’m a digital kind of guy. So my egg-drop competition entries were overly complicated, my solar ovens were a perfect fit for a raw diet, my matchstick suspension bridges were unsafe at any speed, and my analog-circuit-based room-alarm systems would go off at random times in the middle of the night, or not at all, but at least would consistently end up blowing out the LED indicator (what do you mean you can’t connect the power source straight to the LED?)

I might have given up on tinkering, were it not for software… that was something else.

When my father brought home our first computer, a Thomson MO5, I was hooked. I spent hours transcribing BASIC programs from the 3 magazines I could find on the topic (this was Paris, France, not exactly Silicon Valley.) My dad took me to the office so I could talk to some Thomson engineers and debug my floppy disk drive. Later came the TO7, and eventually the Apple IIGS, my first “major” Pascal program to help my mother schedule carpooling (and my first taste of how hard it is to write a scheduling algorithm), my second “major” Pascal program to manage the Prom guest list. I wrote my final Geography report using a page-layout program on the Apple IIGS that probably cost me hours of extra time because of its bugs and the work-arounds I had to find, and got a worse grade for it because “not everyone can afford such fancy software, so we took off a couple of points” (for those of you still confused, THAT is socialism.) Not long after that I was applying to MIT and tinkering with one of the first e-commerce web sites. I love what I do, but would I have discovered this love without those first few lines of BASIC on that MO5 computer, written without anyone’s permission or knowledge?

Over time, though, I have become a little bit complacent about openness. I own an iPhone, and I’ve bought a few apps. I bought music on iTunes, and figured the DRM was not so problematic. I got a Kindle and bought some books. And then one day Apple’s DRM server went down and I couldn’t play music for a few hours. And Amazon decided to recall the book “1984”. And Apple decided to retroactively remove a bunch of apps they considered “not useful enough.” So I started thinking, maybe it’s time to get a different phone.

But I can’t. See, in the interim, I got unexpectedly locked in. I sync my calendar via MobileMe. I sync my music/TV shows via iTunes. Moving to something like a Palm Pre is going to take a significant effort. So how much worse will it be if I get an iPad, get some apps, and Apple decides to change the rules in a way that I don’t like? How locked in will I be then?

This change is happening gradually. At no point are you going to be shocked by an unfortunate Apple decision. You’ll enjoy your iPad, you’ll buy more apps, you’ll enjoy it even more. Apple will make a few decisions that inconvenience you, but you’ll deal. Until one day you’re inconvenienced enough that you might begin to look elsewhere. But you won’t be able to, because your data will be locked in. 3 years ago, we didn’t even have 3rd-party apps on the iPhone. Today, we have more than 100,000, and they’re all rushing to the iPad at warp speed. Change is happening.

One last point. A few months ago, I became a father. My wonderful little boy has an incredible appetite for life. Will he be a tinkerer? I don’t know, but if I had to bet I’d say yes. Will I be able to do for him what my father did for me? What will he tinker with, if everything in the house is a polished, professional, touch-but-don’t-tinker device? If he is to be a maker, a tinkerer, will he be able to fully explore his ideas if the rules of his digital universe are decided by the whims of Apple, Facebook, and Google?

I’m not sure. Maybe he will find a way, the way that kids do. Or maybe we, the generation that is witnessing this change, need to make sure that the rules of computing do not become a permanent, universal, inescapable sandbox.

The Great Content Lockdown of 2010

I had an invigorating and thought-provoking chat with my good friend Oliver Roup today. We agreed that the Apple iPad is going to be an unbelievable success. I’ve thought from day one that it would be huge, but I think it will be bigger than huge. Before the end of the summer, millions of people will own one. Content producers, looking for a way to make money, will flock to it. A virtuous circle will be created. More users. More content. More users. More content.

And so, while killing Flash with one hand, Apple may put a dent in the Open Web with the other. Because if the content producers suddenly have, at their disposal (as Oliver put it) the ultimate platform with identity, micro-payments, a gorgeous interface, and automatic DRM, used by millions of people, why would they continue to funnel millions into their open-web efforts?

Put it all together, and we may begin to see the Great Content Lockdown of 2010. The best, most usable way to read online content will be via iPad apps. Copy-and-paste? Probably disabled. Share via Google Buzz? Only if Google pays to be featured in the “Wall Street Journal app.” In other words, the Web as a platform goes away: there is one client for a given type of content, and it behaves only in the way the producer of this content expects it to. Mashups? Unexpected, serendipitous combinations of features and data? Not likely.

Steve Jobs said the iPad is the most important thing he’s ever done. I think that’s true. I’m just not sure it’s the best thing he’s ever done.

a prediction regarding the Apple “Tablet”

Why a prediction? Eh, cause it’s fun and cause I think the Apple Tablet will have a large impact on consumer computing.

I think Apple will launch a tablet computer in January that will be aimed at saving TV and print journalism. On-demand video and on-demand print magazines and newspapers will be at the forefront. And because those industries want Digital Rights Management, the Tablet will run the iPhone OS so that only approved apps can be installed. It will be great, and the “App Store” concept will continue to rock the house.

In the meantime, Zittrain’s Future of the Internet will be one gigantic step closer, with consumer computing devices tightly controlled by one benevolent dictator. For most people, this will be a very good thing. For innovation, this will be a very bad thing. But it may take a while before people miss it. After all, did people miss Skype before they ever knew it was possible?

Happy 2010, and here’s to hoping we can come up with safe and generative software platforms.

The erosion of our expectation of autonomy, and the Kindle Pledge

As much as DRM bothers me, I’ve tolerated some implementations of it, specifically Apple iTunes, Apple’s iPhone App Store, and the Amazon Kindle, because I’ve gotten more value than pain out of them. And, usually, the DRM didn’t get in the way. But the slippery slope of DRM has reached a dangerous point with the made-for-blog-headlines Amazon story of the last few days.

This weekend, Amazon surreptitiously deleted all copies of Orwell’s “1984” from every Kindle, because of … well it doesn’t really matter, does it? Users legitimately bought a book in a store operated by Amazon, and a few days later it was gone. Jonathan Zittrain saw this coming. Ed Felten makes a solid point that a central issue is transparency, which is very insightful. I’m not sure if it’s the central issue, though. Consider the way the Kindle blog covered the story:

For example, the Electronic Frontier Foundation believes the agreement doesn’t leave room for Amazon to take away already purchased books. Such speculation is most likely pointless, however, since the illegality of the Orwell books distributed probably skirts around the terms of use and Amazon wouldn’t have done the deletions if their lawyers hadn’t cleared it in the first place.

Ah, the lawyers said it was ok, so it must be ok.

The central issue here is, in my opinion, the erosion of our expectation of autonomy. We’re becoming accustomed to fewer and fewer rights. Of course you can’t fast-forward through the FBI warning of a DVD, that operation is “not allowed.” Of course songs can’t be shared on more than 5 ipods. Of course you can’t lend your Kindle book to someone else. Of course Apple can prevent your iPhone from running an app that competes with one of their built-in apps. Of course Amazon can delete a book if someone on their end screwed up, never mind the poor high-school kid who stored all of his school report notes that were then deleted when the book was yanked.

At the core of this erosion of autonomy is a shift in ownership. When the Amazon team revoked the book from every Kindle, they certainly didn’t think it was comparable to walking into someone’s house and burning down their books. Because, from their point of view, the Kindle is within their circle of control. The Kindle is not autonomous. It’s not yours, and its content is not yours. Because if it were yours, then wouldn’t their action indeed be the equivalent of breaking into your house in the middle of the night and burning down that book (maybe they would only do that to a certain Bradbury book)? Ed Felten is right that this is a breach in customer expectation, that Amazon was not transparent about their power over your Kindle. But even if they had been transparent, there would still be this problem of autonomy: Amazon behaves as if they own the Kindle and everything on it. Just read their statement on this issue:

We are changing our systems so that in the future we will not remove books from customers’ devices in these circumstances.

in these circumstances? What about in other circumstances? How precisely is Amazon defining “these circumstances?”

It’s time to take a stand against this. In addition to the price of the Kindle, I paid for each book I downloaded. I want a statement from Amazon that clearly states:

  1. once I’ve purchased a book, it is mine in exactly the same sense as a physical book is mine: Amazon cannot peer into my Kindle and remove data, ever, without a search warrant or similar level of authority.
  2. a solution on transferring of a book from one Kindle to another is in the works.

That’s my Kindle Pledge: until the above two points are addressed, I’m not buying another book from the Amazon Kindle store let alone a second Kindle, even if mine breaks.

UPDATE: Jeff Bezos apologizes for the way this situation was handled. Okay, that’s pretty good (not quite good enough for me to revoke my Kindle Pledge, though). If you read the comments, it’s clear that some people think book deletion was completely legitimate… once again showing how our expectations of autonomy have changed significantly.