Baldur Bjarnason

It's time to treat ebook developers as developers

– Baldur Bjarnason
One of the biggest surprises to those of us who come to ebook development from the web development side of things is the scarcity of documentation.

For years we’ve come to rely on the extensive documentation and information provided by each browser vendor and to come from that to a field where documentation is either non-existent or hidden behind non-disclosure agreements is a massive culture shock.

More than that, the situation prevents us from doing our jobs and holds back ebooks as a business.

The situation

The best ebook developers get today are a hodgepodge of ‘best practices’ guides whose sections on CSS consist mostly of don’t-do-thats, really-don’t-do-thats, and don’t-even-think-about-doing-thats. The only contributions that come even close to qualifying as documentation are Apple’s secret iBookstore Asset Guide and Amazon’s Kindle Publishing Guidelines.

Apple’s guide is limited to only those registered to publish with them directly, which requires both a U.S. tax number and an ISBN number and is only released under a non-disclosure agreement. (Secret documentation. How very unhelpful. Note that Apple even publishes public iOS developer docs, even though you can’t sell iOS apps without their permission. iBooks, on the other hand, which will load any ePub you throw at it, gets a secret, non-public guide. The assumption that no publisher will ever hire an outside party to work on their ebooks is a curious one to say the least.)

And both Apple’s and Amazon’s guide are insufficient as documentation. They are useful guides, but they do not help developers solve any of the problems that come up, every day, when putting together ebooks.

The problems lax documentation causes now

The biggest problem is that it prevents us from solving problems.

When developers have the task of implementing a specific design feature they can solve it in two ways. They can just solve the problem in one platform first then test it on others. If it doesn’t work on one of the other platforms, the developer fixes it, and has to go back and test it in all of the platforms again. And then finds out it breaks somewhere else. This can go on and on and is one of the most expensive, labour-intensive, ways of solving the problem.

It also has the added curse of being completely unrealistic in the ereader market. When developing for web browsers most of the major browsers are freely available (Microsoft even offers freely downloadable OS virtual images for people to test Internet Explorer) and the devices are, for the most part, internationally available. But when developing ebooks there is no simple way, for example, for a non-U.S. developer to test on a Nook or in the Nook app. Where Mobile Safari developers could resort to the iPhone Simulator that comes with Xcode (it’s inside the Xcode app package in the latest version, for those who are curious) the only way you can run iBooks is by owning an iOS device. Also, loading ebooks onto the devices is a much lengthier and involved process than simply pressing the refresh button in a browser. (Or, as I do, invoking the magical incantation that is the keyboard shortcut.)

I’m based in the UK and there’s no way for me to make sure that the ebooks I work on function on a Nook. I can’t test. I can’t look up any documentation. I’m working blind. (And the possibility of B&N rolling out UK support doesn’t solve the problem. That still leaves out all of the other countries. And until ereader vendors offer some sort of developer discount program they really can’t expect ebook developers to own anything more than an iPad and a Kindle. But that’s a subject for another day.)

Something like Kindle Previewer mitigates the problem, but that doesn’t address the basic fact that testing doesn’t belong in the initial stage of solving the design problem.

An example from web development: A problem I tackled last week while putting together a Wordpress theme was solving some issues with the spacing between the widgets I was using. The ideal solution was to use the nth-child selector. (Also documented by Microsoft, for example.)

Now, rather than implement the entire design around this solution first and then go and test it in four versions of Internet Explorer, two versions of Chrome, two of Opera, one Firefox, two Mobile Safaris, and one Desktop Safari (that’s just for starters) again and again and again as I tackled the problem, I went and checked the documentation websites to see what browsers support the selector and in what versions.

I found out that it is well supported by most browsers but was only recently supported in IE, but, since I know that older versions of IE won’t have that big a share of that website’s traffic, I felt that it was okay to if the widgets looked a bit wonky and uneven for those few. (Opera has similar version overview, but with slightly less detail. Still useful because you can get more detail by clicking down to see exactly what each version of their rendering engine supports.)

Then I go and implement it, iteratively, using a couple of browsers, their CSS/DOM inspectors, and a good text editor (I alternate between BBedit and Coda, myself).

Then I go and test it in browsers with usually few surprises because I had the browser’s capabilities in mind when I worked.

Unless I hit a bug, of course, which Chrome, by the way, seems full of these days.

(Public bug trackers for apps like this are a vital tool for developers and an essential part of allowing constructive communication between devs and browser vendors. Being able to go to Chrome’s bug tracker and find out that the rendering glitches are a Chrome bug and not caused by a mistake on my part saves a lot of time and prevents a lot of second-guessing. Public bug trackers for web browsers are incredibly useful to web developers. The same would apply to ereaders and ebook developers. But that’s a subject for another day.)

This process of tackling a design problem is impossible in ebooks. There’s no way for me to check what CSS features the Nook or Kobo iOS apps, Android apps, or devices support. There’s no way for me to discover in what versions of their apps or devices support for a CSS feature was introduced, so I can’t make any estimates about how many readers my choices are likely to affect.

Ambiguity is intolerable

Current documentation, what little exists, is too ambiguous, too sparse in useful detail.

An example of intolerable ambiguity: Amazon’s guide says that the Fire supports the display CSS property.

Fine, but what values? Does it support inline-block? list-item? table? table-header-group? There’s no way to know except by, again, testing. Which is difficult since the Kindle Fire is a U.S.-only product at the moment. Multiply this by a dozen and you begin to see the extent of the problem.

Ambiguous lists like this are useless.

Compare and contrast with how the display property is documented by the browser vendors: Microsoft, Mozilla, Apple.

What developers need are websites, not PDF documents. Websites where we can search for a CSS property, Javascript API, or HTML element, and get a page back that tells us whether it is supported, how it is supported, in what apps or devices, and in what versions of those apps and devices.

This information needs to be public, so third party websites can aggregate them. It needs to be detailed and unambiguous so that publishers don’t have to rely on extensive testing as a part of the initial design phase of their work and keep it in the testing phase where it belongs.

The problems it will cause in the future

The problems lax documentation causes today, when the capabilities of ereaders are so bland they’d make web designers from the year 1997 yawn from boredom, will multiply as ereaders migrate towards more complete web standards support.

As design capabilities improve, the design problems developers will have to solve will become even more complex. Iterative testing as a basic design problem-solving methodology doesn’t scale when you have dozens, if not hundreds, of declarations interacting to create the effect and no way of knowing which one is causing it to fail on a device.

It’s very easy to run into a situation where you know the design problem you are solving and know several solutions that should work according to the spec.

In this circumstance it will be very easy to implement a solution to a design problem, discover that it doesn’t work in a major, essential, platform, and to not have any way of discovering why it isn’t working.

The process of altering the ePub file, loading it into the app, and seeing if it works, is too lengthy and time-consuming for testing to work as a method of discovering exactly what CSS property or value the ereader isn’t supporting and is causing the design to fail.

A problem that’s compounded by the fact that none of these devices or ereader apps offer developer features of any kind, not even remote debugging.

(CSS/DOM inspectors, consoles, debuggers and the like are vital development tools that make great, but otherwise impossible, designs possible and difficult designs easy. Working without them is a bit like working blindfolded. But that’s a subject for another day.)

You don’t test across all target platforms as a part of your basic design and problem-solving methodology. You first build an implementation that is standards-compliant and are reasonably certain should work across your target platforms. Then you test the implementation for unexpected interactions and bugs.

(You of course break the design down into components, individual units that you can implement and test individually. Waiting until you have a complete project before you test is suicide.)

This is before we even consider the issue of bugs. Assuming that all of the implementations are bug-free, ebook developers will still run into this problem because implementations will, inevitably, roll out their support at differing speeds. That is, not all ereaders and ereader apps will support the same things at the same time.

The problem-solving approach ebook developers have been using to date (try, test, try, test, try, test, repeat until insane or successful) simply won’t hack it once the platforms they’re working on have evolved into fully-fledged CSS3/HTML5 web-like platforms.

It won’t be more difficult. It will be impossible. Ebook developers quite simply won’t be able to come even close to matching what their web dev brethren are accomplishing in web browsers. We need documentation for this to work.

The spec is not enough

For all the detail format specifications offer, they are still full of ambiguities. They are also not written in a style that lends itself to being scanned when you are researching a problem.

Just stating that your ereader supports this or that feature of CSS is ambiguous as hell. Do you support it fully? In what versions? All of the values? Or, just the ones you thought people would use?

Spec writers need to write in a specific style to prevent implementation conflicts. They also allow room for variations within compliance with a multitude of shoulds and should-nots. There is scope for considerable difference between two fully-compliant implementations of a spec.

Documentation needs to be written in simpler, slightly less dry, language and with less ambiguity. “This is exactly what the ereader supports and in these versions, the exact syntax, supported values, and maybe a couple of examples.”

The task of writing this can’t be shuffled onto the shoulders of spec writers. Their task is complicated enough to begin with.

Implementations vary, even those that share code

The documentation problem cannot be shared or obviated by relying on a shared rendering engine.

The following browsers are all built on WebKit: Chrome, Safari, Mobile Safari, Android Browser (several conflicting and awful versions), Symbian browser, Samsung Bada browser, Blackberry 6 browser, Amazon’s Silk.

That’s not an exhaustive list. Documentation on Mobile Safari is useless when you’re dealing with the Android Browser.

The features supported and how they are rendered vary enormously across this range. Even in a hypothetical future where ePub readers are all built on only one or two rendering engines, there will still be a need for documentation from each and every vendor.

Feature and rendering differences do not come solely from the underlying rendering engine, but also from the capabilities of the OS, policies of the chosen platform, and the resources and priorities of the vendor.

iOS apps must comply with Apple’s policies. Rendering engines can’t perform feats the OS isn’t capable of. Those same engines need to be adapted to work on varying devices and screens and the capabilities. A hardware-accelerated engine (as most of them are these days) has to be adapted to various graphics hardware.

A lot of the differences, in the WebKit world at least, come from differing schedules. One vendor ships an app and updates it several times, using versions X, Y, and Z of the rendering engine, but in the meantime, hindered by platform issues, another vendor only manages to release an app that uses version X.

These are legitimate and understandable issues, but they need to be documented. Two vendors who both use, for example, a future version of Readium, still need to document what their apps support because the versions they use won’t be the same and the underlying platforms will have different capabilities.

Each and every vendor needs to provide documentation.

The minimum that needs to be done

This is an opportunity.

The transition to newer, more capable, ebook formats like ePub3 and KF8 is the ideal time to begin to build new documentation efforts. The limited capabilities of older formats and older rendering engines mitigate the need for documentation. You can’t do much of anything anyway.

I think everybody would understand if rendering engines based on ePub2 and mobi were allowed to slip, relatively undocumented as they are, into oblivion as their market share fades.

ePub3 and KF8 are opportunities to begin anew and to do it properly this time. Vendors should consider documentation a big part of the move to more capable rendering engines.

The goal should be to:

We need to be able to see, not only what features are supported, but also in what versions of your app or device. Preserving and presenting information about your app’s historical support is just as important as giving us access to documentation on the current version.

Without documentation, most of the capabilities of the modern web formats won’t be used. Without it, the investment that vendors have put into ePub3 and KF8 will be largely wasted. Ereaders might, technically, match the capabilities of web browsers, but using those capabilities will be beyond the ken and resource constraints of most publishers and ebook developers.

And here’s the good news:

You can start small.

Start off with a developer weblog where your engineers write about the features as they add them. (Like IEblog, Surfin’ Safari or Mozilla Hacks.) You can reuse a lot of what you write there in your documentation later on. And a regularly updated developer blog takes a lot of the time pressure off delivering proper documentation.

That is, ebook developers can live with using the information from your developer blogs for a while as you take some time to do proper documentation.

It’s also good PR, if you care about that sort of thing. Double-plus good is if you put together a dedicated developer relations team while you’re at it.

A developer weblog is also the ideal place to publish detailed release notes, something that will help us keep track of what each version of your ereader supports until you get around to doing that documentation site of yours.

The following would also be hugely helpful:

Even though testing shouldn’t be a part of the design process, iterative previews of you work, should. Accurate previews as we work are immensely useful. Best is if we can integrate it into our workflow. (Look at how Coda integrates a web preview, for example. You work on a file. You click preview. You see what it looks like. Imagine working on an html chapter in an ebook, clicking preview, and seeing an accurate preview of that chapter. Bonus points if the preview has a DOM/CSS inspector.)

And, saving the most important, most useful, biggest productivity-boosting suggestion for the last:

Even if your platform won’t support javascript, a CSS inspector has become an essential part of the web development toolkit. I’d even seriously consider giving up documentation in exchange for a CSS inspector if I had to choose. It’s that useful.

Addendum

Both ereader vendors and publishers need to consider where to find talent. A lot of publishing industry veterans are doing a heroic job of re-skilling, but the simplest, most obvious source of web development talent are, well, web developers.

Asking these people to come over into publishing to work for, usually, lower wages is bad enough. Asking them to give up all of their tools, all of their documentation, and give them nothing in return is a deal-breaker.

And the flow will go in the other direction as well. Once people in the publishing industry have trained themselves in the arts of web development and once they realise they could earn more elsewhere, with less frustration and more creativity, they will be changing careers.

Ereader vendors need to, at the very least, match web browsers in terms of tools, communication, and documentation for developers.

If they don’t, the ensuing publishing industry talent crunch will become legendary.

Resources

Browser developer blogs

Browser CSS references

HTML references

Browser web developer centres

Browser developer tools