AI & Heuristics — Promises and Dangers
Things to have in mind if you want to use ChatGPT for usability evaluation
I recently came across this interesting article by Jakob Nielsen on Substack so decided to make a quick TL/DR and also add my personal reflections at the end. In the article Nielsen, share some insights and data about using ChaGPT-4 for heuristic evaluations.
The Baymard Institute uploaded several images (12 images from different websites) to ChatGPT and asked the following question: “What UX improvements can be made to this page?” As you can see in the next lines, the results were not very encouraging… Anyway, they are interesting as they show the current potential and limitations of using this technology.
Here’s a summary to save you a few minutes of your valuable time:
1. UX Reviews: ChatGPT-4 offers an interesting alternative for quick UX reviews, allowing users to send screenshots and request suggestions for improvement, however, in the current state of technology most results were not satisfactory when evaluated by a senior professional.
2. GPT Design Recommendations: Only 19% of ChatGPT-4 recommendations were considered valuable for UX redesign, with a significant portion (72%) seen as a “waste of time”.
3. Bad Suggestions: Approximately 9% of GPT recommendations were potentially harmful, potentially making usability worse if not reviewed by an expert.
4. Expert Assessment is Vital: The value of GPT-4 recommendations depends on the professional’s ability to distinguish good suggestions from bad ones — experience and knowledge in the area are essential.
5. Blind Spots: ChatGPT-4 only identified 24% of usability issues in screenshots, leaving out 76% of other issues found by a professional. GPT is currently unable to evaluate issues related to navigation between screens, which are vital for a good user experience.
6. Evolving UX: Limiting GPT on older training data can lead to outdated recommendations as some UX guidelines evolve over the years.
7. ROI calculations: The study suggests that AI can save time in identifying usability issues, but costs additional time in rejecting less useful suggestions, resulting in an overall negative ROI. (however, it is worth remembering that this technology evolves very quickly).
8. The Role of AI in UX: Although ChatGPT-4 may not be ideal for identifying usability issues, it has great value in generating ideas and assisting with UX research tasks, improving and positively impacting the workflow of the UX teams.
Conclusions & personal insights
ChatGPT-4 shows promise, but it still has several limitations, making it more suitable for generating ideas and assisting in UX research rather than as the sole tool for uncovering usability issues. The future of AI in the UX process has potential for improvement, and it’s crucial to remember that today’s AI is the starting point on a journey toward more capable tools. (The AI we have today is the worst we will ever have). Combining AI with the expertise of a human professional remains essential and ideal for success in UX design and evaluation.
AI should be your co-pilot, not your God who always brings universal truths.
Don’t worry, AI still won’t take over your job, but a professional who uses AI as a co-pilot can take over from those who don’t use it (improvements in productivity and results).
AI is a path of no return.
AI is not a trend or fad.
Trends come and go.
AI is here to stay.
🏄 Don’t fight the tide, adopt, surf.
More about me
https://linktr.ee/faleco
Recommended Reading
https://www.uxtigers.com/post/ai-new-ui-paradigm
Sources
https://jakobnielsenphd.substack.com/p/ai-ux-evaluation
https://baymard.com/blog/gpt-ux-audit
Did you enjoy reading? Share it (if you want to) 🖤
