Dr GPT or: How I Learned to Stop Worrying But Hate the AI Part 2

Part 2: A Painful Second Shot

ChatGPT for Writing Code

In my last post on my profile, I explored BingAI and concluded it was a long way from being a useful tool for writing actuarial reports. In Part 2, I’ll be looking at how Chat GPT performs at writing code.

While the external marketing for Bing AI emphasizes its web surfing and analysis abilities, the marketing for ChatGPT, especially GPT4 emphasises its coding and analysis skills. One oft-quoted statistic from the recent announcement of GPT4 is being able to pass the US bar exam in the 90th Percentile of students. In theory, a tool with this level of analytical skill could be a massive boon for actuaries. This could lead to large amounts of tedium and repetitive tasks being automated, freeing up actuaries to give their time and thoughts towards decision-makingand high-level analysis.

Initial attempt at GPT3.5

As an arbitrary starting point, I provided GPT3.5 with a triangle (or structured aggregated set) of claims and asked it to estimate the associated claims reserves.  I attempted this a few times and quickly noticed a pattern forming. It intuitively adopted a standard actuarial technique (the chain-ladder method) but at some point before projection to ultimate, started malfunctioning and either gave no answer, or like Bing AI earlier, confidently gave an incorrect answer.

Initial codewriting with GPT4

I then moved on to attempting the same task with GPT4. At first, it told me it didn’t have enough space to calculate the projection; a perplexing response given that the method isn’t space intensive. However, I was encouraged that it appeared to have an awareness of its potential limitations - something I had not yet encountered in my interactions with AI. It then gave a comprehensive breakdown of the methodology for using the chain ladder method to calculate claims reserves.

I asked it to write some code using the open-source statistical software package, R, for me and my excitement soon turned to frustration. While it knew its own space limitations earlier, it seemed completely unaware of them now. It gave me truncated code when it ran out of memory. When informed that the code was truncated, it apologised and produced the exact same half written code. Eventually after multiple messages of negotiation I got it to write the code across several messages.

The (many) issues with the code

It wrote me well formatted code with explanations for every step. The issue came when I tried to run it. I soon discovered where Bing had inherited its ability to spew convincing nonsense. Predictably, the code did not work at all. It was on the right track and then gave me multiple made-up functions to try. It seemed as if it was guessing how a programming language should work as opposed to how R works in reality.

Conclusion on GPT4 for coding

I would have no problem with it informing me that it couldn’t complete this task as it had insufficient information about R. This was not the case. I told it every issue that the code was having and every time it apologised and responded with further nonsense.

I was thoroughly disappointed with this as I had expected code writing to be a strong point of GPT4. This process took me multiple hours, due to GPT4’s glacial response speed before I gave up because it had yielded not a single estimate of any use.

My overall conclusion is that even if ChatGPT is useful for writing general code, or even high-level “conceptual” programme structures, as a time-saving tool for the intricate grunt-work and for specific and niche use cases (which most actuarial cases will fall under), GPT4 is far from a useful tool.