Letter to My Two Sons – November 13th, 2015

[this is a little bit raw… on purpose.]

My sons,

You are just 6 and 3, and so you don’t know what happened tonight. A group of suicide bombers killed 150 people in Paris, your father’s hometown. The feeling in my gut today is much like the one I felt on that Tuesday in September 2001, as I tried to get to my office in TriBeCa, shell-shocked people on the street walking past me, thousands of dead in the rubble. Profound sadness, deep anger, frustration, and powerlessness. And this nagging feeling that one of the victims could, under slightly different circumstances, have been me or… you.

That day in 2001, I got to the office just a few blocks north of the towers, just an hour or two after they’d collapsed. I logged into one of our web servers, found an unused IP address (that’s how we did it back then, kids), and built a manual list of “people I know are safe in NYC” (a poor man’s Facebook Safety Check). I frantically emailed friends and built up the list. The URL went around to a few dozen people. A few friends and friends of friends found each other and, hopefully, a small measure of relief. In retrospect, I realize I was coping by doing the only thing I knew how to do: contribute a small positive on a day of pure horror. I don’t mean to praise myself, I simply did what all decent people did that day: help any way I knew how. I knew HTML and web servers, and so that’s what I did.

Much will be written about today, November 13th 2015. Extremists on the right will embrace confirmation bias and recommend closing borders, arming the public, and generally distrusting brown people. Extremists on the left will also embrace confirmation bias and lay the blame entirely on the West’s foreign policy.

To be honest, I don’t really know what to think. Well, no, that’s not quite true: I think those extremists on the right (including many presidential candidates today) are idiots, maniacs, and shouldn’t be allowed within spitting distance of the seat of power. They stoke the fires of retaliation and intolerance, feeding on fear to push their agenda, the furthest thing from democracy and freedom. So yeah, I guess on some level, I do know what to think.

That said… might it help to fight at the source those who committed these awful acts so they don’t get the chance to do it again? Maybe. On the flip side, did we do things that others saw as acts of aggression, for which they then retaliated? Maybe that’s part of it. Are there suicidal/homicidal maniacs who will use anything as an excuse to hurt innocents? Probably. I don’t really know for sure.

So what do we do?

If there is one thing I hope to teach you, it is this: you will not always be safe. It kills me to say this, because I am biologically wired to protect you, and yet… You shouldn’t live your life seeking safety at all costs. You shouldn’t compromise your own freedom because madmen took lives, even if it’s dozens, hundreds or thousands. You shouldn’t compromise your own freedom the second, third, and fourth time something terrible happens, either.

What you can do is choose to be one of those people who help. One of those people who make the world better, in small or big ways. You will live through many more terror attacks, stupid governments, unnecessary wars. The human condition is, in many ways, heartbreaking. You cannot make the heartbreak go away. But you can choose to be a positive force. You can choose to be a helper. Even if it’s something as small as writing a bit of HTML by hand on a warm Tuesday in September, tears streaming down your face, because it’s the only thing you know how to do and because maybe, maybe, it will help one person.

the responsibility we have as software engineers

I had the chance to chat this week with the very awesome Kate Heddleston who mentioned that she’s been thinking a lot about the ethics of being a software engineer, something she just spoke about at PyCon Sweden. It brought me back to a post I wrote a few years ago, where I said:

There’s this continued and surprisingly widespread delusion that technology is somehow neutral, that moral decisions are for other people to make. But that’s just not true. Lessig taught me (and a generation of other technologists) that Code is Law


In 2008, the world turned against bankers, because many profited by exploiting their expertise in a rapidly accelerating field (financial instruments) over others’ ignorance of even basic concepts (adjustable-rate mortgages). How long before we software engineers find our profession in a similar position? How long will we shield ourselves from the responsibility we have, as experts in the field much like experts in any other field, to guide others to make the best decision for them?

Well, I think that time has come.

Everyone uses software, very few people understand it. What seems obvious to a small elite group is completely opaque to the majority of the world. This gap is incredibly hard for us, the software engineering elite, to see. A few examples:

  • The Radiolab Podcast did a wonderful piece – Trust Engineers – where they explored the case of Facebook running experiments on its newsfeed. For non-engineers, there’s an incredible feeling of breached trust upon realizing that a set of flesh-and-blood humans have that much control over the algorithm that feeds them daily information. (And, for that matter, to most researchers used to interacting with an IRB, there’s complete shock at what Facebook did.) For most engineers, including a number of very good and ethical people at Facebook, it’s surprising that this is even an issue.
  • A couple of years ago, a friend of a friend – who happens to be a world-renowned physician and research scientist – asked me: “Ben, can the system administrators at work read my email? Even if they don’t have my password?” The answer is yes and yes. This is obvious to us engineers, so much so that we don’t even think twice about it. To a non-engineer, even an incredibly smart person, this is absolutely non-obvious.
  • A close friend, another very smart person, was discussing something with his young child recently, and I overheard “if you don’t know, ask the computer, the computer knows and it’s always right.” Where do I begin?

We, software engineers, have superpowers most people don’t remotely understand. The trust society places in us is growing so rapidly that the only thing that looks even remotely similar is the trust placed in doctors. Except, most people have a pretty good idea of the trust they’re placing in their doctor, while they have almost no idea that every time they install an app, enter some personal data, or share a private thought in a private electronic conversation, they’re trusting a set of software engineers who have very little in the form of ethical guidelines.

Where’s our Hippocratic Oath, our “First, Do No Harm?”

I try very hard to think about this in my own work, and I try to share this sense of duty with every engineer I mentor and interact with. Still, I don’t have a good answer to the core question. Yet it feels increasingly urgent and important for us to figure this out.


This week, I joined Clever as VP Engineering. Clever makes K-12 education software vastly more efficient and effective by simplifying how students and teachers log in. It’s this simple: imagine if you could give teachers and students 10-15 minutes back in every single class. That’s 30-40% more time for actual teaching and learning. That’s what Clever does today, with much more in the works.

I’m incredibly excited about this new adventure, and I want to gush a bit.


My priorities in work are:

  1. people
  2. mission
  3. product

People – strong contributors who know how to work in teams that accomplish more than the sum of their parts – are my top priority. A mission and a product, no matter how good, survive contact with the real world only if backed by strong, honest team players.

A Mission – a clear goal to make a positive, socially beneficial dent in the universe – is my close second priority. Products come and go, pivots happen, but a strong mission gives an organization an invariant, a true north when the storm hits.

And finally, a strong Product. Because once you have great people, and once you have a clear and stable mission, you still need a compelling product to deliver that mission to the market. The product is the last mile of your impact on the world.


Clever meets all three of my priorities in spades.

I am blown away by the quality of people at Clever, starting with co-founders Tyler, Dan, and Rafael, and including every engineer, business-development partner, school experience advocate, recruiter, etc. Clever team members have oodles of education experience, with ex-teachers, ex-DOE, and ex-school-technologists joining hands to build the next-generation education platform. And it’s not just individual quality and skills, as Clever has also built a strong team culture, one that mirrors the values of the education products we want to see. Every team member is always a student, and Clever is a group effort.

The Clever mission is clear and deeply impactful: to save teacher and students time, all the while protecting student privacy and preserving data access controls enforced by schools.

And finally, the Clever product is catching on like wildfire. There remains a mountain of work because there’s a mountain of opportunity to make education software better. But the market is already speaking, and Clever has struck a a very clear chord.

Join Us

We’re a growing team of 20’ish engineers, passionate about applying technology to making K-12 education far more effective. We’re committing to building a diverse team, because teams with a variety of life experiences build better products, and because darnit it’s the right thing to do.

Do you like working on code that makes the world a better place? Do you want to learn from and teach your teammates every day? Do you want to work on hard technical problems not just because they’re hard, but because they’re hard and impactful?

Clever’s a unique place and a unique opportunity. Use your powers for good. Send me a note.

(your) information wants to be free – obamacare edition

My friends over at EFF just revealed that Healthcare.gov is sending personal data to dozens of tracking sites:

It’s especially troubling that the U.S. government is sending personal information to commercial companies on a website that’s touted as the place for people to obtain health care coverage. Even more troubling is the potential for companies like Doubleclick, Google, Twitter, Yahoo, and others to associate this data with a person’s actual identity.

The referenced AP story uses even more damning language:

The government’s health insurance website is quietly sending consumers’ personal data to private companies that specialize in advertising and analyzing Internet data for performance and marketing, The Associated Press has learned.

Sounds pretty bad, right? Except it’s almost certainly not what it sounds like. It’s almost certainly a simple mistake.

How could this be a mistake, you ask? Here’s what almost certainly happened:

  1. Someone at Healthcare.gov wanted to analyze patterns of usage of the site. This is often done to optimize sites for better usage. So they added a tracker to their page for MixPanel, for Optimizely, for Google Analytics, and a couple of other sites that help you understand how people use your site. In all likelihood, different departments added different trackers, each for their own purposes, almost certainly with good intentions of making the web site more usable.
  2. Meanwhile, someone else responsible for social media of HealthCare.gov added a “Tweet This” button, and someone else added a YouTube video. Once again, these come in the form of widgets, often snippets of JavaScript code, that load resources from their respective home base.
  3. Separately, someone built the web form that lets you enter basic information about yourself so you can find a health plan. That information is, in large part, fairly personal: your age, your zip code, whether or not you smoke, etc. And for some reason, almost certainly completely random, they used a web form with an action type of GET.
  4. Here’s the first mildly technical point. When you submit a GET form, the data in the form is appended to the URL, like so:

    Not a big deal, since that data is going to Healthcare.gov anyways.

  5. And now for the second mildly technical point. For tracking purposes, trackers often blindly copy the current URL and send it to their homebase, so that the trackers can tell you users spent 5s on this page, then 10s on that page, etc. In addition, when your browser requests an embedded YouTube video, or an embedded tracker, it sends the current URL as part of the request in a so-called Referrer field.
  6. Put those two technical points together, and boom: a web site that collects personal information with GET forms and uses third-party tracking widgets tends to send form data to those third parties.

This is extremely common. Many web sites with sufficiently large engineering teams have no idea how many trackers they’ve embedded. It’s typical for a web site to move from one site analysis tool to another and to forget to remove the first tracking widget in the process. When the Wall Street Journal reported on these issues a couple of years ago with their fantastic What They Know series, they forgot to mention that their own page has a half-dozen trackers embedded.

I’ve said it before, and I’ll say it again: unfortunately, your information wants to be free. My favorite analogy remains:

when building a skyscraper, workers are constantly fighting gravity. One moment of inattention, and a steel beam can fall from the 50th floor, turning a small oversight into a tragedy. The same goes for software systems and data breaches. The natural state of data is to be copied, logged, transmitted, stored, and stored again. It takes constant fighting and vigilance to prevent that breach. It takes privacy and security engineering.

So, am I letting Healthcare.gov off the hook? Not at all, they should have done their due diligence and done a more thorough privacy audit. And using GET forms is particularly sloppy, since it leads to data sprayed all over the place in logs, referrers, etc.

But was this a deliberate attempt at sharing private data with private companies? Not a chance. The press should do a better job of reporting this stuff. And, to my wonderful friends at EFF, this is a gentle nudge to say: so should you. It’s important to differentiate between negligence and malice, to not spread fear, uncertainty, and doubt, even when it’s issues we care about.

The good news is that HealthCare.gov has already responded by (a) reducing their number of trackers significantly and (b) submitting form data using XMLHttpRequest or POST. The bad news is how many people now actually believe that this was intentional, conspiratorial data selling. If that was Healthcare.gov’s intentions, there are much sneakier ways of doing that without getting caught so easily.

Oh, and if you want to understand more about trackers and block them as you surf the web, try the very excellent Ghostery extension for your browser.

managing photos and videos

This holiday, I finally spent time digging into how I manage photos and videos. With 2 young kids and some remote family and friends, this requires a good bit of thinking and planning. I know I’m not the only one, so I figured documenting where I landed might be useful to others.

I started with Dave Liggat’s Robust Photo Workflow, and found much of it resonates with my needs. Here’s where I landed:

  1. I take photos with a DSLR and two phones. My wife takes photos with her phone. We both take videos with our phones. We use Dropbox/Carousel auto-upload, which works just fine on both iOS and Android. For the DSLR, I manually load photos over USB.
  2. All photos and videos are now available on my desktop Mac (via USB or Dropbox). When I’m ready to review/edit photos, I drag and drop the batch into an all-photos/ directory I keep within my Dropbox.
  3. Hazel automatically categorizes photos and videos into subdirectories of the form 2015/01/. It’s really kind of awesome.
  4. all-photos and all-videos are thus simple date-classified folders of all of my photos and videos. They’re backed up locally using Time Machine. They’re backed up to the network using Dropbox. I can imagine eventually snapshotting this to Amazon S3/Glacier, but right now that doesn’t feel too urgent.
  5. I use Lightroom5 as an editor only, so if I blow away my Lightroom proprietary catalog info, it’s not that big a deal. To do this, I tell Lightroom to look at photos in all-photos without moving/copying them. After I’ve added a bunch of photos to the all-photos directory by drag-and-drop, I tell Lightroom to synchronize its catalog with the source folder, which takes a few seconds and gives me a screen with my latest imported photos and videos. I can then edit photos, reject them if they’re bad, and write back JPG/XMP data to each photo’s originating directory using Lightroom export. Dropbox backs those up automatically. To remove bad photos (blurry, etc.), I flag them as “rejected” in Lightroom using the X key, and when I’m done I command-delete, which gives me the option of removing the files from disk, too. I do this only for clear rejects, and it makes my mild OCD happy since I know I am not keeping totally useless files around, and the overhead of deleting photos is low. I could also delete photos easily using the Dropbox UI, which is pretty good, and then re-synchronize in Lightroom.
  6. I can then use Carousel (or Dropbox) on any mobile device to browse through all of my photos. It’s surprisingly good at dealing with large photo libraries (I have 20K) and large photos (I have a bunch of 13MP photos). As in, really, really good, even on a puny phone. Better than anything else I’ve seen.
  7. I’ve been using Flickr for years for private photo sharing, and Lightroom is really good at exporting to Flickr. That said, at this point I’m thinking of moving to Dropbox/Carousel based sharing. I can easily bundle photos & videos into albums on Dropbox, whereas videos are still limited on Flickr. Carousel conversations around a few photos are great with family. The only bummer is that Carousel and Dropbox have some mutually exclusive features: albums on Dropbox, conversations on Carousel. I suspect Dropbox will fix that in the next year.
  8. What I’d love to see:
    • unified photo features in Dropbox and Carousel
    • export Dropbox albums as directories of symlinks in my Dropbox folder, and export Carousel conversations in some other file-based way, too.
    • Lightroom export compatibility with Dropbox/Carousel albums.

I’m super happy with this new process: one funnel, easy, low overhead, and a very solid long-term photo storage solution. I’m only relying on RAW/JPG files and directories of said files to be readable for the long term, and that seems pretty safe. Lightroom is awesome, but I could replace it with a different tool if I needed to.

One more thing: if you’re going to use Dropbox to store all of your photos, make sure you pick a strong password and set up 2-factor authentication.

Power & Accountability

So there’s this hot new app called Secret. The app is really clever: it prompts you to share secrets, and it sends those secrets to your social circle. It doesn’t identify you directly to your friends. Instead, it tells readers that this secret was written by one of their friends without identifying which one. The popularity of the app appears to be off the charts, with significant venture-capital investment in a short period of time. There are amazing stories of people seeking out emotional support on Secret, and awful stories of bullying that have caused significant uproar. Secret has recently released features aimed at curbing bullying.

My sense is that the commentary to date is missing the mark. There’s talk of the danger of anonymous speech. Even the founders of Secret talk about their app like it’s anonymous speech:

“Anonymity is a really powerful thing, and with that power comes great responsibility. Figuring out these issues is the key to our long-term success, but it’s a hard, hard problem and we are doing the best we can.”

And this is certainly true: we’ve known for a while that anonymous speech can reveal the worst in people. But that’s not what we’re dealing with here. Posts on Secret are not anonymous. Posts on Secret are guaranteed to be authored by one of your friends. That guarantee is enabled and relayed by the Secret platform. That’s a very different beast than anonymity.

In general, if you seek good behavior, Power and Accountability need to be connected: the more Power you give someone, the more you hold them Accountable. Anonymity can be dangerous because it removes Accountability. That said, anonymity also removes some Power: if you’re not signing your name to your statement, it carries less weight. With Secret, Accountability is absent, just like with anonymous speech, but the power of identified speech remains in full force. That leads to amazing positive experiences: people can share thoughts of suicide with friends who can help, all under the cloak of group-anonymity that is both protecting and empowering. And it leads to disastrous power granted to bullies attacking their victims with the full force of speaking with authority – the bully is one of their friends! – while carrying zero accountability. That kind of power is likely to produce more bullies, too.

This is so much more potent that anonymity. And if this fascinating experiment is to do more good than harm, it will need to seriously push the envelope on systems for Accountability that are on par with the power Secret grants.

Here’s a free idea, straight out of crypto land. In cryptographic protocols that combine a need for good behavior with privacy/anonymity protections, there is often a trigger where bad behavior removes the anonymity shield. What if Secret revealed the identity of those users found to be in repeated violation of a code of good behavior? Would the threat of potential shame keep people in line, leaving the good uses intact while disincentivizing the destructive ones?

where the system eats itself

Larry Lessig just launched MayOne.us, the SuperPAC to end all SuperPACs. The idea is disarmingly simple: since SuperPACs funded by billionaires are corrupting politics, let’s crowd-source a SuperPAC funded by individuals, which will then work to put in power officials who answer to the people, maybe by undoing the whole SuperPAC insanity. Use a SuperPAC to kill all SuperPACs.

This is a fascinating pattern that we’ve actually seen before. And it makes me very, very happy, because it is the ultimate policy hack.

Take the GPL or the Apache License, two significant software licenses that make possible open-source and thus much of today’s software. These licenses, especially the GPL, enforce certain constraints on how source code can be used. If you take GPL source code, modify it, and redistribute it, you have to provide the source code to your modifications, too. If you don’t… you lose your license on the code to begin with, and now you’re guilty of copyright violation because you redistributed code without permission.

In other words, many open-source licenses work only because they leverage strong copyright law. The same goes for Creative Commons: you can freely license your work while requiring that people give you credit for it, or, if you prefer, only if they use it for non-commercial purposes. The only reason you can add those constraints is because Creative Commons is layered on top of Copyright. Some people believe Copyright Law has overreached. Those same people are using that Copyright overreach as a foundation for a stronger Commons.

The GPL, Creative Commons, and now MayOne basically use the overreach of the system against itself.

I find this idea – that an unnaturally strong system of rules can be counter-balanced by using the system against itself – fascinating and kind of awesome. Are there other areas where this can be applied?

But before I digress: go pledge to MayOne.us. Let’s make SuperPACs eat themselves.

Obama lets NTSB exploit widespread tire vulnerabilities

Stepping into a heated debate within the nation’s transportation safety agencies, President Obama has decided that when the National Transportation Safety Board (NTSB) discovers major flaws in transportation equipment, it should — in most circumstances — reveal them to assure that they will be fixed, rather than keep mum so that the flaws can be used in espionage, senior administration officials said Saturday.

But Mr. Obama carved a broad exception for “a clear national security or law enforcement need,” the officials said, a loophole that is likely to allow the NTSB to continue to exploit safety flaws to apprehend terrorist suspects as they go about their daily routines such as driving.

On Friday, the White House denied that it had any prior knowledge of the AirBleed defect, a newly discovered safety vulnerability in commonly used Michelin tires that often leads to spontaneous tire explosion when driving for prolonged periods of time at exactly 44mph. This flaw has led many manufacturers to urgently recall cars and many drivers to behave erratically in 45mph speed zones. The White House statement said that when such flaws are discovered, there is now a “bias” in the government to share that knowledge with car and tire manufacturers, so a remedy can be created and distributed to industry and consumers.

Sources indicate that some senior officials had urged the NTSB to get out of the business of weakening commercial transportation systems or trying to build in “trapdoor failures” that would make it far easier for the agency to intercept suspected terrorists. These officials concluded that the practice would undercut trust in the American auto industry. In recent months, Detroit has urged the United States to abandon such practices, while Germany and Japan, among other nations, have said they were considering pulling all auto production facilities back to their own countries.

Not surprisingly, officials at the NTSB and at its military partner, the United States Special Operations Command, warned that giving up the capability to exploit undisclosed safety flaws in widespread commercial equipment would amount to “unilateral disarmament” — a phrase taken from the battles over whether and how far to cut America’s nuclear arsenal.

When interviewed at his home in Maryland, John Smith, head of the NTSB in the 1980s, appeared incredulous: “Do you mean to tell me that the American Government knew about widespread life-threatening safety issues in our cars and chose not to disclose those findings to its citizens, just in case this weakness could be used to go after a presumed terrorist? I don’t believe it. The US government serves its people first, and never in a conspiracy-theorist’s wildest dreams would they engage in such despicable behavior.”

[a light parody of this New York Times Article.]

to Brendan and Mozilla

I was in the middle of writing a blog post about the controversy surrounding Mozilla when my Twitter feed exploded with the news that Brendan Eich stepped down from his new appointment as Mozilla CEO. So this is a different post. Also, this is not a post about Prop8 (which I abhored) or gay marriage (which I consider a basic civil right.)

to Brendan

There is little love lost between me and Brendan. We have different styles, and I butted heads with him in tech discussions on a regular basis while I was at Mozilla. He was, at times, infuriating. To be honest, he drove me up the wall.

Yet through all of these disagreements and conflicts, one thing was never in doubt: Brendan has the Mozilla mission of inclusiveness, openness, and freedom engrained in his heart. This is a man who has tirelessly worked in the open for close to 20 years, giving up riches at other companies so he could create an open playground for the world to use, a counter-balance to corporate interests on the Web, and opportunity for billions, even if they never know his name.

So Brendan, thank you, for everything you’ve done for the Open Web and for Humanity. The world is in your debt. You can take the man out of Mozilla, but you can never take the Mozilla mission out of the man.

to my friends at Mozilla

Some people love Mozilla for its mission, but much of the world doesn’t get it. People use Firefox because they like it; others use a different browser because they like that one better. I don’t think that will ever really change. It’s easy to become very depressed about this, to think “how can people be so harsh on Mozilla when we’ve been such good citizens?” or “can’t people see that Firefox is the only browser built by a non-profit?” People don’t care as much as they should. One bad PR cycle might be enough for them to switch browsers.

I want to suggest that this apathy can be empowering: don’t count on others groking the mission. Follow the Mozilla mission on your own terms, because you know it’s the right thing to do. Do the right thing because it is the right thing.

Keep doing the right thing, friends, and be excellent to each other.