Notes / Converting Japanese EPUBs for Kindle

I use a Kindle Oasis (9th Generation) for reading light novels, but the ecosystem is very closed, so I need to convert books before I can sideload them. There isn't much structure to these notes, just some things I wanted to write down.

See the kindle-tools repo.

Why a Kindle?

I did want to get a Kobo because of the EPUB support and general more open ecosystem but went for a Kindle instead.

I had two criteria

How well text selection works for Japanese, i.e. when you long press on a word does it select that word?
Can it somehow load EPWINGs? (A format for Japanese dictionaries.)

For the first point

I had seen how text selection behaves on a kindle and it worked okayish/good.
I couldn't find any information about how it behaves on a Kobo, so this was left as unknown.

For the second point

For Kindle can use epwing2kindle. Even better there are lots of precompiled monolingual dictionaries from How to Use a Kindle to Learn Japanese (Matt vs. Japan). There is also jmdict-kindle.
Also should be possible for Kobo, see this reddit thread. The initial steps are the same as for the Kindle, but then use Penelope.

Basically, something closed that I knew would work (and had almost all of the work already done) vs. something more open, but unknown.

I also got a Kindle Oasis very cheaply on eBay.

My old Kobo

Just writing some things I tried with my old Kobo Touch N905B. I updated to a newer (Sep 2019) firmware, but it still couldn't handle Japanese books. This is almost certainly due to how old the model is, even though they seem to release the same firmware for all of them.

I even installed KOReader on it (almost like a custom firmware), but while it is very easy to load custom dictionaries in the StarDict format (see here), using them on Japanese books is extremely painful. It also couldn't handle furigana. There is no long press to select a word function, you have to manually drag the ends every time, which is error prone. However this also could be due to the age of the model. KOReader does look really good but I think it still needs some work for what I want to do with it. Would be interesting to see if Yomichan could be ported to it, then it would have by far the best dictionary experience available.

TODO: See how a new/recent Kobo behaves, and also how KOReader behaves on it.

Since it doesn't support KF8, it can be used to view the MOBI7 fallback in a dual MOBI file, this is useful for testing.

Converting EPUBs

There are three ways to do this

KindleGen - An older tool by Amazon (now discontinued), most recent build I could find was from 2014, see here. It produces a dual MOBI, containing both a fallback MOBI7 and a KF8. By default it embeds the source EPUB into the built MOBI, so file size can balloon, but it is possible to disable this with the -dont_append_source option. Some more information about it here.
Kindle Previewer - The continuation of KindleGen, seems to be built on top of it too. But no builds for Linux.
calibre - Open source, can directly produce a KF8/AZW3 file. However, it messes up formatting, in particular margins for vertical text. None of the margin settings helped, an idea might be to open up a kindlegen output book and see what the styles are like compared to the calibre output (see KindleUnpack).

So KindleGen was the only option for me.

Here are some output file sizes when run with different options (the EPUB I tested it on had images as well as text).

original 13.4 MB
kindlegen -> 29.7 MB
kindlegen -c0 -> 30.2 MB
kindlegen -c1 -> 29.7 MB
kindlegen -c2 -> 29.6 MB

kindlegen -dont_append_source -> 16.3 MB
kindlegen -dont_append_source -c0 -> 16.8 MB
kindlegen -dont_append_source -c1 -> 16.3 MB
kindlegen -dont_append_source -c2 -> 16.1 MB

NOTE: Run without -c2 first to see if there are any errors, because the compression can take a little time. It also doesn't save much space, so might not be worth it anyway.

Errors with `toc.ncx`

A lot of the EPUBs I ran kindlegen on failed to convert because of errors with toc.ncx.

Error(prcgen):E24001: The table of content could not be built.
...
Info(prcgen):I1041: The file format version is V8
Info(prcgen):I1031: Saving PRC file
Info(prcgen):I1032: PRC built successfully
Info(prcgen):I1038: MOBI file could not be generated because of errors!

I opened them in the editor in calibre and deleted toc.ncx, it's not required for EPUB3, see this, and this. Note simply unzipping the EPUB and deleting it will leave a broken EPUB since it is referenced elsewhere in the EPUB.

TODO: Write a script to properly strip toc.ncx, and add it to the repo.

Stripping the MOBI7 in the built dual MOBI

It is also possible to strip the MOBI7 in the dual MOBI to save space, see here. This really isn't worth doing, the savings are small. However I did add it to the repo as kf8-only.

original 13.4 MB
kindlegen -dont_append_source -c2 -> kf8-only -> 16.0 MB

Some pictures

A book with horizontal text formatted with calibre, this looks okay.

A book with vertical text formatted with calibre, notice the text selectors have the wrong orientation.

A book with vertical text formatted with calibre, the text is squished right up to the edges of the screen, none of margin settings in calibre fixed this for me. The margin settings on kindle affect vertical text in the vertical direction, so not helpful for this.