Dr GPT or: How I Learned to Stop Worrying but Hate the AI

Part 1: A Frustrating Start

An unavoidable future?

“AI is transforming actuarial practice in many ways. Actuaries who embrace AI will be able to leverage its benefits while mitigating its risks. Actuaries who ignore or resist AI will risk becoming obsolete or irrelevant.” These words are directly from Bing AI when asked how AI can transform the actuarial world. Do these words ring hollow or are we on the precipice of a massive shakeup in how we work? That’s what I set out to find out.

I ventured to answer this question by testing the capabilities of Bing AI (which is still in Beta) and ChatGPT. Bing AI is a generative AI that can search the web and perform certain functions, most notably, generating text. ChatGPT is based on similar technology but cannot access the internet.

This blog exclusively tests the capabilities of Bing AI to write and analyse reports. Further blogs on ChatGPT are coming soon.

Test 1: High hopes for an ambitious task

A great deal of the marketing for Bing AI focuses on its analytical capabilities. To test this, I tried to see if Bing could test if a Part VII Report (an actuarial report for non-actuarial readers) written by one of my colleagues, was following regulatory guidelines.

This is where I came across the first major issue and biggest talent of Bing AI: its ability to fluently and convincingly lie. Upon asking for a summary of a report link I sent it; Bing AI fabricated an entire summary of a report from a Big 4 Firm. I told it about the mistake, it apologised and provided me with an entirely new and equally fake report. This report came from an arbitrary consultancy, although one that worked on Part VII reports in the past.

I finally gave in and asked it if it could just open the link. At this point, Bing AI informed me that it didn’t have the capability to open links, despite asking me to send links multiple times. Overall, a frustrating experience.

Test 2: Lower hopes for a less ambitious task

My next tactic was to inform it where it could find the report and ask it to summarise the report. At first this seemed a lot more effective with only one mistake in its description of the report: the page count. I asked it how it came to that page count and the link to the report it was reading. Its response was to send me a link to a document that, upon further inspection, does not exist. It’s quite clear from this that its priority is to provide believable as opposed to accurate answers.

Conclusion

To conclude, Bing has capabilities which possibly in the future, will be able to assist actuaries with report analysis. However, at this point in development, for these tasks, it is fundamentally useless and unknowingly incompetent. Its ability to lie and fabricate is so endemic, that no work it produced for me could be trusted. While progress is still being made, there is a long way to go until this program can be relied upon to do anything of utility.

The question remains however as to whether the issues are due to a lack of available data or due to the fundamental problem of there being no algorithm for truth.

While my unfortunate conclusion for today is that Bing AI is worse than useless for all the tasks described above, come back soon for my thoughts on ChatGPT3.5 and ChatGPT4 as tools for reserving in “Part II: a Frustrating Second Shot”.