One of my reasons for blogging has been to expose my ideas to external scrutiny. It was to get feedback, so I could improve my model of the world.

One of my biggest surprises was that I didn’t receive a lot of feedback.

There were times when I received general, non-specific, positive comments. Some people praised specific articles in generic ways. A handful of times I received an email from someone questioning a specific statement.

There was one article that I received some feedback that a particular article didn’t seem to put me in a very flattering light.

However, the feedback was sparse and it wasn’t detailed.

One conclusion I could reach is that I didn’t get much feedback because my writing is awesome, and my arguments are tight.

However, I think it’s a simple matter of incentives. If I read, listen to, or watch something online, where is my incentive to reach out to the author and make my points in a tactful way? What’s the point?!

As time has gone on, I think that’s the bigger reason.

Enter ChatGPT and LLMs

I’ve found plenty of practical applications for ChatGPT and its ilk since they have become available.

One of the most useful applications is for receiving feedback in relation to articles I’ve written. The feedback I’ve received is better than I’ve received from anyone, at any time, about anything on the blog.

Below is an example of a prompt I use with ChatGPT or Claude+ (via Poe):

You are a group of highly intelligent individuals with diverse experiences and well-read backgrounds, and have been asked to review some text for a friend who publishes articles on a personal blog which is not for a large audience, but for what he describes as ‘kith, kin, and kindred spirits’. Communicate your thoughts in a helpful, thoughtful, candid, tactful, encouraging way. Assume he is highly educated, well-read, and intelligent, and is in his early- to mid-forties. Analyze the following text, and respond to each of these requests separately. For each response, start with a heading, then follow it with two blank lines (similar to if you wrote \ n \ n in html), and then your paragraph response. The tasks are: Summarise the text (SUMMARY); assess its coherence and whether its reasoning is sound (COHERENCE & REASONING); fact-check claims, and/or list some statements that could be fact-checked in bullet-point format (FACT-CHECK); identify any weak, questionable, or biased parts OF THE TEXT that might be noticed by keen observers, and provide a strong critique of the arguments that are made (CRITIQUE); generate creative and memorable metaphors or analogies that might emphasise the points of the text, and list these as bullet-points (METAPHORS & ANALOGIES); recommend articles, books, videos, podcast episodes, or other resources that touch on the the topics of this text, if relevant, and suggest other people or organisations who have covered similar ground, and list these in bullet-point format (ideally, around 5 or 6 suggestions) (OTHER RESOURCES).

After this prompt, I’ll copy and paste the draft text and the feedback is pretty astonishing. For reference, I’ve included the feedback I received based on the first draft of this article at the foot of this article.

I’ve even gone one further, and with the help of ChatGPT, created a Python script that speeds this process up further. (The script is at the bottom of this article, in case you’re interested: you’ll need an API key from OpenAI to get it working.) When I run this script (which I can do with the push of a button on my Stream Deck (a keyboard-like thingy attached to my computer)), I can execute the prompt above based on whatever is in the clipboard of my computer. It accesses the OpenAI API for ChatGPT-3.5-turbo and then opens Notepad and spits out the response in text, without having to go into my browser. (I’ve created other scripts too, that simply summarise what’s in my clipboard, or do a decent job of redacting what is in my clipboard.)

It blows my mind.

I can go even further. Via Poe, I have access to Anthropic’s Claude model that has a 100,000 token window. That means it can keep approximately 75,000 words in memory. I’ve been able to copy and paste huge amounts of my written text, and interrogate the model for all sorts of insights about my own writing (and thinking process).

Eventually, I’m going to go even further and fine-tune an OpenAI GPT model on basically the entire corpus of writing on my blog, and various other mediums. This will give me an objective perspective of my writing – including its strengths and foibles. It’s unlikely to cost me more than $20. And I will feel comfortable asking all sorts of weird and wonderful questions I wouldn’t have wanted to bother another human being with.

I acknowledge that this type of feedback might not be “better” than the sort of feedback I could get from an intelligent, thoughtful reader who had the incentives, time, and ability to communicate feedback with me. But frankly, that’s not available at the moment. Even feedback from any one person is going to have its own idiosyncrasies. I’ll take whatever feedback I can get, ideally from multiple models, and integrate it as much as I can! It’s a useful tool that wasn’t there before.

Feedback for the first draft is below. I added the final paragraph (directly above) based on this feedback.

Feedback from ChatGPT-4:

SUMMARY

The author expresses that the initial motive behind blogging was to expose his ideas to external scrutiny and improve his worldview based on the feedback received. However, he observes a scarcity of feedback and speculates that the lack of incentives for the readers might be the main reason. To overcome this, the author has started to leverage AI technologies, particularly language models like ChatGPT and Claude+, to review and provide feedback on his writing. He is astonished by the results and finds them more beneficial than human feedback. Further, he has automated the feedback process using a Python script and plans to fine-tune these models on his entire blog to gain a comprehensive and objective perspective of his writing style and thought process.

COHERENCE & REASONING

The author presents a clear and logical narrative. His initial disappointment from the lack of feedback, to his speculation about the reason (lack of incentives for readers), followed by his discovery and utilization of AI language models for feedback are coherently linked. His reasoning is sound, given that the incentive to provide detailed, constructive feedback is often missing in the context of online content consumption. Moreover, his rationale for using AI tools for feedback - their availability, convenience, and non-judgmental nature - is plausible and compelling.

FACT-CHECK

  • Claim: ChatGPT and Claude+ are useful tools for generating feedback on writing. Fact-check: As of my knowledge cutoff in September 2021, both ChatGPT and Claude+ were known to provide text generation and analysis, making them potentially useful tools for such a purpose.
  • Claim: Anthropic’s Claude model has a 100,000 token window and can keep approximately 75,000 words in memory. Fact-check: As of my last training data in September 2021, I cannot verify this claim.
  • Claim: Fine-tuning an OpenAI GPT model on his entire blog corpus is likely to cost less than $20. Fact-check: As of September 2021, the cost of fine-tuning a GPT model would depend on several factors including the size of the dataset, the training time, and the specific costs associated with using the necessary computational resources. It’s uncertain if this could be achieved for under $20.

CRITIQUE

While the author makes a compelling case for using AI language models as a source of feedback, the argument largely assumes that AI-generated feedback is inherently superior to human feedback. However, this assumption can be critiqued. Firstly, human feedback, while sparse and non-detailed as the author found, brings a unique element of personal connection, real-world context, and empathetic understanding which AI might not replicate fully. Secondly, the objectivity of AI can be a double-edged sword. While it provides non-biased feedback, it might also miss out on the nuances, emotional tones, and cultural contexts which humans naturally infer. The author could have elaborated on these limitations.

METAPHORS & ANALOGIES

  • AI as a personal writing coach: just as a coach provides objective feedback and guides an athlete towards better performance, AI provides unbiased critique and suggestions for improving one’s writing.
  • Fine-tuning the AI on the author’s blog corpus can be compared to training a dog to understand and respond to its owner’s specific commands.

OTHER RESOURCES

  • “Superintelligence: Paths, Dangers, Strategies” by Nick Bostrom discusses AI and its potential impact on society, including its use in domains like content creation and analysis.
  • “Writing Tools: 55 Essential Strategies for Every Writer” by Roy Peter Clark - This book could help understand the fundamentals of good writing, against which one can compare AI-generated feedback.
  • “On Writing: A Memoir of the Craft” by Stephen King - This book provides insights into the writing

Feedback from Claude+:

SUMMARY

\n \n

The author has started blogging to receive feedback to improve their thinking and writing. However, they have received little feedback over the years, likely due to a lack of incentive for readers to provide feedback. Recently, the author has found AI models such as ChatGPT and Claude+ useful for providing detailed feedback on draft texts. The author describes using prompts to ask the models to analyse and critique their writing, and has even created a script to automatically provide feedback. The author finds the quality of feedback from the AI models ‘astonishing’ and mind-blowing, and plans to fine-tune a model on their full blog corpus to gain further insights.

COHERENCE & REASONING

\n\n

The passages flow cohesively from the author’s initial aim of gaining feedback through blogging, to their disappointment at receiving little, to their recent discovery of AI models as a source of high-quality feedback. The reasoning - that AI models have greater capability and incentive to provide feedback than humans in this context - seems sound. However, it is arguable whether feedback from AI models can adequately replace human feedback, as models may lack certain qualities like emotional intelligence or life experiences that shape human judgment and opinions. The intended audience seems suited to the level of technical explanation around the AI models.

FACT-CHECK

•The author started blogging to get feedback and improve their thinking, but received little feedback from readers over time.

•The author has recently found AI models like ChatGPT and Claude+ useful for providing detailed feedback on their writing.

•The author created a script to automatically get feedback from ChatGPT-3.5-turbo.

•The author plans to fine-tune an OpenAI GPT model on their blog content.

•The author finds the feedback from AI models ‘astonishing’ and ‘mind-blowing’.

CRITIQUE

\n\n

The passage presents the use of AI writing feedback models in an overwhelmingly positive light. A more balanced analysis could consider potential downsides, risks or limitations of relying on AI feedback versus human feedback. For example, models may provide superficial or inaccurate feedback, lack emotional intelligence, or be unable to gauge how a text might impact different audiences. The passage could also expand on exactly what kinds of insights the models have provided, with examples, to enable the reader to better judge their value. A discussion of any efforts made to verify or compare the AI feedback against human feedback would also strengthen the argument being made.

METAPHORS & ANALOGIES

• AI writing assistants as 24/7 writing coaches.

• Models as tireless editors providing customized feedback.

• Script as a virtual feedback machine.

• Fine-tuned model as personalised writing advisor.

OTHER RESOURCES

•The AI Literacy Workbook by Kai Eckert

•Drafts podcast episode on AI writing feedback tools

•Articles by Anthropic and OpenAI on feedback model developments

•Talks from RISE conference on responsible AI

•Blog posts byMichael Casey on automated writing assessments

•Resources on overcoming writer’s block by using AI feedback (e.g. Writer’s Block Solved course)

•Related thinkers: Sam Harris, Yuval Noah Harari