September 14, 2010

Personal essay: How do I create CCLaP's ebooks?

(Every day, I like to post at least a thousand words of original content to the CCLaP website; on the days I don't have a review of a book or movie ready, I thought I would try other material, such as this series of personal essays, looking at a topic in the arts from my life that I think you might find relevant or entertaining too. You can click here for a master list of all personal essays now written, if you're interested.)

I recently heard from an acquaintance of mine, Texas filmmaker Christopher Sharpe, asking for some advice on what kinds of ebook software I use for CCLaP and how I go about actually making the center's publications; and that reminded me that I've been receiving so many questions like this in the last year or two, I've been meaning to sit down and simply write it all out for public consumption once and for all. So thanks for finally pushing me to do so, Chris! Although before anything else, let me caution people that the latest news concerning these subjects continues to change seemingly every week, and that if you're coming across this article in the future, there could be big sections of it that are now out of date; and that of course is simply the nature of emergent technology, and why it is that so many people need advice on the topic in the first place.

So first, for those who are still confused, let me make it clear that an "ebook" is just another type of digital file, much like how images might be swapped between devices as JPEGs or a song downloaded as an MP3; but unfortunately, one of the problems that plagued the ebook industry for decades was that none of these software companies could agree on a common tech standard, so all invented their own that will work on only their devices. And indeed, although this situation has gotten a lot better just in the last year (but more on this in a bit), for an electronic publisher to really target as many potential readers as possible, it's important not just to offer their books in the popular formats about to be discussed, but also such second-tier specs as .PDB (originally designed by Palm, a company that went out of business earlier this year, but with plenty of Palm devices still floating around out there), .LIT (The Windows Mobile format, which like many Microsoft experiments never really caught on widely), and .MOBI (the official tech standard of Mobipocket devices, a sort of also-ran ebook company but, much like Blackberry, still with a small but passionate group of customers).

Typical PDF page

Like I said, though, in just the last year the ebook industry has really started consolidating itself into a mere three main choices, which if you offer them yourself will safely cover 90 percent of your total customer base; and the irony is that one of these formats is pretty much the oldest commercial option for ebooks out there, the .PDF format by Adobe, also known as "Acrobat" files become of the application that used to be needed to make and read them. Originally designed as an experimental way for lawyers to safely transmit legally binding files electronically, a PDF file essentially takes a "photograph" of each page of your document; thus, the finished manuscript looks exactly on the other end like it does on your own computer, good for exacting designs like legal documents and professionally designed books, but with your computer interpreting these pages as images instead of text, and so unable to do things like elegantly rewrap the document's text to the size of a different screen (er, kinda...but again, more on that in a bit).

Typical EPUB page

The other major format, then, and the one looking likely to become the overwhelming industry standard in just another few years, is the newly invented .EPUB format; designed precisely from the ground-up as an open-sourced, non-competetive way to share electronic books across devices and platforms, in just the last year it's been intensely embraced by Apple (for their iPhones and iPads), Sony (for their "Reader" e-ink devices), Barnes & Noble (for their "Nook" e-ink devices), and more. And the reason that EPUB files transfer so well from device to device is that they're nothing more than the same HTML and CSS documents that make up a typical website as well, only bundled in a fancy way into an EPUB "container" that most end-users aren't even aware of, thus using a simplified markup system that a lot of designers and publishers are already familiar with, and that was designed from its beginning to be able to be re-wrapped to fit the dimensions of whatever screen you're using, as well as to be able to include images, hyperlinks, hierarchal style specs and more.

Typical TXT page

So with all these choices at your disposal, and all these different formats to output at the end, let me urge all authors and publishers to start this entire process by thinking of their book in terms of an official "master" copy at its core, one saved in good ol' plain ASCII text (or in other words the .TXT format), and with all other versions being mere temporary spinoffs of that master; so in other words, if you find that a change needs to be made to the book, the best option is to literally throw away all other copies besides the .TXT master, make the change to that master, then output all the PDFs and EPUBs and MOBIs and whatnot again. Because seriously, take it from me, from years and years of having to learn this lesson the hard way over and over, if you have seven versions of your book and then suddenly realize that you have three typos to correct, it's absolutely guaranteed that at some point you will lose track of which versions have had which corrections made and which haven't, and eventually will find yourself with seven completely different versions of your book, none of them exactly correct, much like literally seven different monks copying out a Bible in the Middle Ages by hand and all of them accidentally inserting their own unique series of mistakes. And this can be maddening, a lot more maddening than the minute or two it takes to output the entire file again in a new format from scratch; and in the meanwhile, you have the comfort of knowing that you have a clean archival copy of your work for posterity's sake, one free of all the junky special formatting automatically inserted by Microsoft Word and other word-processing programs, a format that literally hasn't changed since World War Two and that can be easily opened error-free in everything from smartphones to old '50s supercomputers to freaking Java-equipped kitchen appliances. (Just what the world's been waiting for -- a text-to-speech poetry reading by a robotic refrigerator.) And saving a file in .TXT format couldn't be easier either; most word processors have it simply as an option in their "Save As" menu, or you can make the process even easier by getting into the habit of doing all your first-draft writing in one of the many "Simple Text" type applications that come free with virtually all operating systems on the market now.

Typical Microsoft Word page

So once you do have your clean, pristine ASCII-text version of your book, where do you go from there? Well, since it's the easiest conversion, I usually like to start with the PDF; because like I said, since an Acrobat document is essentially a "photograph" of each page, you actually create these pages in whatever page-layout software you prefer, sending the entire thing at the end not to a paper printer but rather the PDF "virtual printer" option included now in both the Mac and Windows operating systems. (See the printer specs of your operating system for more details; in many cases, it's as simple as choosing the "Send to PDF" or "Print as PDF" option from the "Print..." dropdown menu.) Believe it or not, for example, I create CCLaP's PDFs in plain ol' Microsoft Word, and the covers in Adobe Photoshop, then join them together using Acrobat Pro, a commercial application made by Adobe for doing such fancy things as replacing specific pages in a PDF, fine-tuning permissions, adding metadata and more. (And note that there are a plethora of freeware programs out there as well for accomplishing such tasks, some that work better than others.) You can design PDFs to be any size, to better fit whatever destination you have in mind -- some publishers, for example, put out PDFs in a 16 by 9 format, specifically for viewing on HDTVs and iPhone screens -- although I in particular design CCLaP's PDFs mostly to be sent to a home laserprinter, for those who dislike reading on computer screens altogether, which is why I make two Acrobat files for each of the center's books, one for Americans that is 8.5 by 11 inches, and another for Europeans designed in the A4 format. (Of course, don't forget before the conversion to re-add all that special formatting that the ASCII version removed, like quotation marks with little curls on their tails, actual copyright symbols, properly sized em dashes, and all the other little changes that these layout programs make when using their "AutoFormat" option.)

Calibre user interface

Creating the EPUB version, then, is both a more complicated endeavor and a simpler one; because like I said, an EPUB file is basically nothing more than a bundle of smaller files that any web designer will already be familiar with, an HTML file for the text and JPEGs for the images and a CSS document to tell ebook devices how exactly to display that text and those images, and those with an inclination can literally make these files by hand if they want, for an ultra-fine-tuned control over how their book looks and feels. (And in fact there are several online resources for teaching budding "ebook programmers" how exactly to do this; just for one example, check out EPUB Zen Garden, created by the same people behind the popular CSS Zen Garden.) I however have preferred so far to use the free application Calibre, a remarkably sophisticated program for being essentially shareware; designed originally as an easy way for e-reader owners to convert books from their original format to whatever suits them best, its EPUB option I've found produces books that work almost perfectly on every device I've now tested. Calibre does a lot more than this, too, by the way, including gathering up the contents of hundreds of different websites each day and then "pushing" them to your device via subscriptions as standalone ebooks; although I'm not affiliated with the development team, I'm a big fan, and really encourage you to patronize them if you have a chance. But that said, there are of course lots of other options out there at your disposal for creating EPUB files, the side effect of it being such an open system; for another example, see the similarly excellent Feedbooks.com, which will convert your manuscript to every major format simultaneously, and automatically add your book to their popular public sharing library if you so choose.

Typical HTML page

Of course, it's important that you choose the right original format from which to do your EPUB conversion, which gets into what I was talking about before; that although technically Calibre will convert from a PDF to an EPUB file, the conversion tends to be riddled with mistakes, with the software regularly doing things like mistaking page numbers for actual embedded story text, screwing up the look of chapter titles, getting freaked out over tables and more. (And this of course is the problem with ebook reading software and devices that promise to "resize" and "reflow" the text inside a PDF automatically for you; that since these are essentially giant images of each page, and these conversion apps are essentially trying to guess at what's important in that image and what isn't, none of them do nearly as good a job as they advertise.) But since EPUBs are basically glorified HTML files, conversion from an actual HTML version of your book is almost flawless (and almost instantaneous in Calibre too), which is the way I highly encourage you to make your own EPUB yourself; if your original document is in Word, WordPerfect, Quark XPress, Pagemaker or InDesign, for example, all of these apps have a simple "HTML" or "Web Page" option in their "Save As..." menus. After adding all the formatting and page breaks that you want in that original application, simply output the manuscript in this HTML format, then open that HTML document in Calibre and convert it to an EPUB. (And don't forget, by the way, that Calibre's wizard-style conversion process makes it easy to add all kinds of other details to the document, including attaching a fancy cover, inputting its metadata, creating a hyperlinked table of contents, fine-tuning tab lengths and more; see the Calibre user guide for a lot more on these subjects. Also don't forget that there's no need to manually add page numbers to an EPUB document; that's part of why people like EPUBs so much, is that it's the ebook reading software itself that will automatically add page numbers for you, ones that match up exactly with the number of pages that particular device parses the book out as, no matter what its screen size.)

Kindle Publisher user interface

So that leaves only one more major option, which of course is the version for Amazon Kindles, which as of autumn 2010 is the undisputed juggernaut of the ebook industry; and as usual with Amazon, creating a Kindle book is a special exception that doesn't relate to even a single bit of anything I just said, because when you have an 80 percent market share you get to make up whatever rules you want, and make your customers jump through however many endless hoops you feel like making them jump through. So in the case of the Kindle, for example, there literally is no such thing as a standalone file for the device that others can swap, buy and sell away from Amazon*; not only is the Kindle ebook format (.AZW) a closed proprietary system that you're not allowed to have access to, but Kindle owners aren't even allowed to add files to their devices unless they've literally purchased them directly from the Amazon Kindle store. So like I said, there are a lot of hoops to jump through to get your own book into the Kindle store; first you need to apply for and be approved as an official seller there, then you have to log in to their online "book creation center," then very carefully copy and paste your book's contents into their WYSIWYG onscreen editing system, literally one chapter and one title at a time, then pray that it all works perfectly or else be forced to start the entire process all over again. And then you have to wait for Amazon to approve the book, then wait some more before the book starts actually showing up in searches, then wait even some more before receiving your first payment, knowing all the while that you will still only receive an average of 50 percent of the cover price of each sale (Amazon keeping the other half for the "privilege" of selling your book at Amazon), and knowing that Amazon can yank your book from its store at any time and for any reason, which by the way will erase every copy out in the physical world that your customers have previously purchased. It's my hope that as the ebook industry expands in the next decade, and Amazon loses the stranglehold it currently has over the format, that they won't be quite so draconian in their policies towards self-publishers and small presses; but as always with Amazon and artist-hostile draconian BS, I ain't exactly holding my breath.

*UPDATE: Many thanks to CCLaP reader Gerard Collins, for letting me know that my information above on Kindles was based on early versions of the device, and that the newest Kindles have a cord and USB slot for connecting to your home computer, just like any other e-ink device. He also let me know that Amazon actually purchased Mobipocket a few years ago, specifically to gain access to their formatting technology (turns out that the .AZW format is merely .MOBI with an extra level of anti-piracy measures added), which means that you too can create .MOBI files that should work just fine on Kindle devices, and that Kindle owners can transfer to their devices directly without needing the Kindle store. The rest of the information above is still correct, as far as I know.

Anyway, I hope this has been a useful if not too general overview of the process; and along those lines, for converting into the more minor formats mentioned at the beginning of this article, you will need to seek elsewhere for the specific instructions, a process far too detailed to be covered in this already too-long tutorial. For additional questions, comments or corrections to anything I've just mentioned, please either leave a comment at CCLaP's Facebook group or write me directly at (cclapcenter [at] gmail.com). Now go get publishing!

Filed by Jason Pettus at 2:10 PM, September 14, 2010. Filed under: Arts news | Design | Literature | Profiles | Reviews |