Book editing with AI: Code Interpreter vs. Claude — there’s no contest

Update July 12, 2023: I thoroughly tested out Claude’s abilities as a content and structural editor and gave it high marks. Read my full review here.

Update July 13, 2023: I tried using Claude as a copyeditor, looking for grammar and punctuation errors. It failed the test, but I found one AI that worked. Read my full article here.

I’ve been anxiously looking forward to the general release of OpenAI’s Code Interpreter plugin for ChatGPT. Why, you might ask? Well, it’s because Code Interpreter can read and analyze large files. Book-length files. I have a lot of book-length files that I want analysed.

For example, I’ve got a series of books set in the Krim virtual world that I’m about to publish on Amazon and I forgot to update my outlines while I was writing them, so I’ve since forgotten what happens in what book, and who all the characters are, and what they look like and sound like.

I could also use some help with structural editing — do all the parts make sense? Are there any loose ends or plot holes? Are the character motivations clear?

Yes, I have a human editor I use for this, but the more stuff I can fix on my own, the closer the book will be to a finished product by the time he looks at it.

Well, ChatGPT’s Code Interpreter beta is out and available to all ChatGPT Plus users — that means, people who pay for the premium version.

To get to it, click on your face at the bottom left of the screen, then select “Settings.” The Code Interpreter is under the “Beta features” tab, right below the “Plugins” option.

I spent quite a bit of time with it yesterday trying it out, and I was extremely disappointed.

Code Interpreter: The bad, the worse, and the reasonably attractive

So, yes, you can now upload a whole book for ChatGPT to read. I uploaded the text of my novel Krim Times by clicking on the little plus icon in the text entry field where you normally type your prompt.

And it turns out that Code Interpreter can read the novel, but only for a particular definition of the word “read.” So, for example, it can find all the capitalized words in the book, or calculate the frequency in which different words appear.

But it doesn’t understand what it reads. So it doesn’t know that “Joe” and “Schmoe” refer to the same person, or that he’s described as having a full head of red hair at the beginning of the book and as being bald a chapter later — and that this is a problem.

I asked it for a list of characters in the book, and the best it could do was find all the capitalized words. It came back with a list of over 200, starting with “Marshal,” “Shoppe,” “Bill,” “Zoning,” “Groups,” “Cyril,” “Krim,” “Thanks,” “Something,” and “Inhaler.” This was a completely useless list.

But it did create an reasonably pretty word cloud of my 25 most-used words:

I have watched a lot of online tutorials about Code Inspector and the various things it can do, and it is an extremely powerful tool — but for data analysis, not for books.

Well, not yet at least.

Claude 2: The good, the great, and the ugly

Today, Anthropic released Claude 2 to the public. It’s completely free, for now at least, and can handle file uploads too. Just go to Claude.ai and sign up. I logged in with Google, so it only took a few seconds.

Then I clicked on the little paperclip icon in the prompt entry field at the bottom of the screen, uploaded my text file, and asked Claude, “What are the main characters in this book?”

Here is its response, unedited:

CLAUDE:

Here is a summary of the key characters and events in the story:

• Marshal Henderson Trask – The head of security for the Krim Chamber of Commerce. He is investigating a series of attacks on tourists and merchants in the city.
• Joe Steelstrikes Phantomblade – Trask’s lieutenant who assists in the investigation.
• Matilda Scarletstrike – A swordfighter hired by the Chamber of Commerce to help Trask and Joe.
• Baron de Mowbray – A land baron who is suspected of being behind the attacks after torturing and killing a newspaper editor. He has an army and a grudge against the local newspaper.
• Seymour Gellhorn – The publisher of the local newspaper AviNewz who has been targeted by the attacks.
• Cyril Booker – Gellhorn’s editor who is kidnapped, tortured and killed by Baron de Mowbray.
• Margrave Ademar – The head of Baron de Mowbray’s army. He is suspected of being the griefer but it turns out not to be him.
• Jerald Rex Crewe – A process server trying to serve subpoenas on the grid. He is kidnapped to keep him quiet.
• William Lockton – An assistant grid manager who is revealed at the end to be the real griefer, trying to drive down property values so an outside group can purchase the grid.

All of these are completely correct.

Then I asked it for ways the book can be improved. It gave me a lost list of things I can work on. A depressingly long list. But one item caught my attention — it said that I needed to make some things clearer. What things? What’s the thing I need to clarify the most?

And it told me.

CLAUDE:

The section that could use the most help with clarity is Chapter 5, where the fire occurs at the newspaper building. There are a few areas in particular that could be clarified:

• • The beginning of the chapter jumps right into the action of the fire but it’s not initially clear whose building is on fire or why it matters. A transition sentence explaining that this is the newspaper office on fire would help for readers who don’t recall that detail from previous chapters.
  • When Seymour runs out screaming “Fire!” it would help to explicitly state that this is Seymour Gellhorn, the newspaper publisher, so his identity is clear. His name isn’t mentioned in this scene so a reader might not make the connection.
  • The mechanics of how the firefighters extinguish the fire could use more explanation. Readers may not understand how a hose connected to a roof pipe puts out a fire inside the building. A sentence or two clarifying the building’s fire suppression system would help.
  • Trask speculates about fires being bad for the grid but his thought process is a bit vague. Spelling out more specifically how fires could hurt tourism, economics, etc. would add important context about motivations.
  • The conversation between Trask and the fire chief about the cause jumps around a bit. Transition language like “After inspecting the scene, the chief concluded…” could help link things more logically.
  • When the chief says “she” might have been delivering papers, it’s unclear who he means. Recapping that a burned body was found would make this clearer.

Even just a few extra sentences in these spots to orient the reader, provide background, and smoothe transitions would really help this chapter flow and increase clarity. Let me know if you need any suggestions for specific text to add to address these issues!

Claude got a few things wrong. Seymour yelled “Fire!” at the end of chapter four, not in chapter five. And it’s Trask who says the line “She might have been delivering the paper,” not the fire chief. And for some reason Claude spelled “smooth” as “smoothe.”

But otherwise, Claude is pretty much spot on.

Here are a few other things I asked it for:

• What are the main themes of the book?
• Please create a scene-by-scene outline of the book.
• What’s the single best thing I can do to make this book more readable?
• Please create a list of all the locations mentioned in the book, as well as their descriptions.
• What is the main character’s character arc?
• Are there any plot holes in the story?
• Are there any places where the character motivations are unclear?
• Please look through the book for grammar problems and identify my most common mistakes.
• Is there anything more I can do to give each character a distinctive voice? For example, for each major character, can you suggest some ways that their dialogue could be more reflective of their personalities and goals?

For example, on that last question above, Claude gave me some excellent and very insightful suggestions on how I can improve dialogue.

CLAUDE:

Here are some suggestions for making the dialogue and speech patterns of the major characters more distinctive to reflect their personalities:

Trask:

• • Use more pauses, stuttering, and filler words like “um” and “uh” to show indecision.
  • Short, choppy sentences to reveal lack of clarity.
  • Focus dialogue on food and complaints about exertion.

Baron de Mowbray:

• • Formal, arrogant tone. Longer, more sophisticated sentence structure.
  • Threatening and aggressive language. Curses, insults, violent metaphors.
  • References to torture, battles, feudal rights.

Seymour Gellhorn:

• • Nervous, fast-paced, rambling way of talking. Run-on sentences.
  • Defensive about criticisms. Makes exaggerated claims.
  • Frequently tries to negotiate or make deals.

This is all useful advice that is perfectly spot-on for those characters.

But now, let’s get to some hard stuff. Grammar.

A month ago, I tested out an earlier version of Claude, and found that it had problems with grammar. (It also cost money, and it had a limit only a 100 questions a month that it could answer.)

This time around, Claude’s grammar was much, much better. In nearly every case it immediately found the problem, and provided a good explanation of the error and how to fix it.

There was one case where it got it wrong, where it didn’t find the error in: “Less than 50 people showed up for the presentation.” I asked it to look at the sentence again, and it apologized, and picked up on the fact that “Less than” should be “Fewer than.”

I’m going to give it a 95 percent score on grammar, much much better than it did a month ago — and roughly comparable to ChatGPT.

The other improvement I’m seeing with Claude today, other than the fact that it’s now free and more widely available, is that it has a nice, searchable chat history and no limits that I could see on how many questions you could ask.

There is a limit on how big a file you can upload. My tenth book in this series is 84,000 words long, and it was over the limit. My ninth book is 59,000 words, and that was fine. So if your book is much longer than 60,000 words you might have to work on it in parts.

Oh, and here’s one very cool thing — by default, Anthropic will not use the text you upload to train its AI.

The only area where ChatGPT’s Code Interpreter wins is in the charts. Claude can only do text charts.

Final showdown

Ability to pull information from the book, like outlines and character summaries:

• ChatGPT’s Code Interpreter: F
• Claude: A

Ability to analyze text and make useful editing suggestions:

• ChatGPT’s Code Interpreter: F
• Claude: A

Ability to create cute charts based on data:

• ChatGPT’s Code Interpreter: A
• Claude: C

You can watch me talk about this on YouTube here:

Maria Korolov

MetaStellar editor and publisher Maria Korolov is a science fiction novelist, writing stories set in a future virtual world. And, during the day, she is an award-winning freelance technology journalist who covers artificial intelligence, cybersecurity and enterprise virtual reality. See her Amazon author page here and follow her on Twitter, Facebook, or LinkedIn, and check out her latest videos on the Maria Korolov YouTube channel. Email her at [email protected]. She is also the editor and publisher of Hypergrid Business, one of the top global sites covering virtual reality.