Can ChatGPT or Claude Underwrite Deals? A Practical Analysis

Artificial Intelligence has been around for decades, but the rapid development of large language models (LLMs) has changed how businesses use AI. LLMs are a type of AI trained on massive amounts of text and data, and they can read, summarize, and reason in ways similar to humans. ChatGPT, Claude, and Gemini are all examples of LLMs. 

Because of these capabilities, LLMs perform many tasks way more efficiently than humans. In finance, people use these tools to read reports, extract data, or summarize documents. Impressively, LLMs can give answers within seconds, with comprehensive, logical, and plausible-sounding explanations.  

However, anyone who has used these tools may have already noticed: LLMs can hallucinate. They fabricate information that is inaccurate or unsupported, which can lead to severe underwriting problems - a small error in revenue and balances can affect a funding decision. 

This article explores how LLMs can help with underwriting and where they can go wrong, along with reasons behind it. At the end, we will suggest a practical way to reduce the risk of hallucinations. 

What LLMs Are Good At 

Before we delve into the downsides, it's undeniable that LLMs can be helpful in certain areas. As LLMs are trained on billions of texts and learn language patterns, summarizing is one of their strengths. Also, when data is explicitly stated and clearly structured in a standardized report, LLMs can extract it with high accuracy. Nevertheless, their accuracy plunges when it requires industry knowledge or financial reasoning (Balsiger et al., 2024).  

• Document summarization: Condense lengthy financial reports, application forms, and business profiles into concise summaries. 

• Explain financial concepts: Answer general questions about underwriting terminology, industry standards, or financial concepts. But accuracy may drop when LLMs aren't trained enough on very niche industries. 

• Extract explicit data: Pull data from standardized and well-structured documents. Reminder: sometimes LLMs mix real numbers with fabricated numbers in the same output. 

• Draft outputs: Generate a risk narrative or summary based on information the underwriter has provided. 

Truth Behind Plausible Answers  

LLMs always sound confident. Even when they don’t get the answer, they may invent a number. Then you might be wondering: Why don’t LLMs simply admit they cannot find the answer? The reason traces back to how they are trained. 

LLMs aren’t originally trained to search for a verified answer. They’re trained to predict the most likely next word. Through this process, they learn language patterns, including that a question is usually followed by an answer. That’s why, even when they can’t find the requested information, they may generate something false instead of saying “not found”. 

Research shows LLMs fall short and hallucinate in both quantitative and qualitative outputs. 

Quantitative Failures: Extracting Numbers from Financial Tables 

Many people think reading a number directly from a table is easy for LLMs, as it is right there on the page. But that’s not always the case. A 2024 study by Balsiger et al. asked GPT-4 and Google BARD to extract financial data from 46 corporate annual reports. Here’s their combined average accuracy: 

From the chart, currency accuracy came in at 98.8%, which is very high because a currency label such as “USD” or “CHF” is typically explicitly stated and requires little interpretation. But accuracy dropped when extraction required financial reasoning or industry knowledge, such as for revenue, EBIT, and gross profit. Researchers found several reasons behind these failures: 

1. Non-standard terminology: Income statements vary across industries. For example, gross profit is a term used by industrial or trading companies and doesn't apply to the income statement of a financial company. That’s where LLMs cound’t find the answer and hallucinated numerical values instead. 

2. Non-standard fiscal years: When a company’s reporting period didn’t run January to December, both models had issues identifying the correct figures, leading to incorrect results. 

3. Magnitude errors: When figures were presented in thousands, both models frequently missed the scale indicator and reported numbers off by a factor of 1,000. 

Qualitative Failures: Understanding Financial Text 

Not only do LLMs struggle to extract numbers accurately, but they also struggle to understand professional texts. In 2024, researchers tested whether GPT-4 could detect contradictions and inconsistencies in real financial reports. They fed the model 30 already-published financial reports and asked it to flag any contradictions it found (Deußer et al., 2024).  

As a result, GPT-4 flagged 260 contradictions. However, when professional PwC auditors reviewed every single output, only 8.84% of the 260 contradictions were confirmed as genuine. In other words, only 1 in 11 flagged contradictions actually existed. 

They found the reasons for incorrect answers are: 

1. Ambiguous wording: GPT-4 identified a contradiction because the financial report itself was worded ambiguously. The researchers still considered these findings relevant, as the wording should ideally be clarified. 

2. Missing context: Without the full context in the prompt, GPT-4 didn’t have the full picture and labeled statements as contradictory. 

3. Knowledge gaps: GPT-4 lacked specific subject knowledge or misunderstood financial terms. So the model might interpret two statements as contradictory when, in fact, they were describing the same thing. 

4. Unusable output: Responses didn’t make sense or lacked proper explanation, rated as unusable by the auditors.  

A Practical Solution 

We’ve seen several reasons why LLMs generate inaccurate answers. When they cannot find an answer or are unsure, they make up assertive statements or numbers to sound convincing. Here’s a simple prompt you can add to help reduce hallucinations from GPT, Claude, or other AI when you need to scrub financial documents: 

Answer only from [documents]. Cite the exact page for every figure. 

If it’s not there, say “not found” or “not sure”. Don’t fill from memory or make up numbers. 

However, this approach only reduces the probability of hallucination. It doesn’t eliminate it. General-purpose LLMs were trained on broad internet text, rather than specifically for financial underwriting. As such, they lack the industry-specific knowledge needed to interpret financial information. 

On the other hand, purpose-built tools are trained and designed for a specific domain. You may have seen many specialized AI tools in the market nowadays, such as AI health monitors, medical imaging, voice agents, etc. You wouldn’t go to GPT or Claude for every specialized task because they’re not trained to be experts in every field.  

The same applies to underwriting deals. Go for a purpose-built tool that is designed and trained to extract accurate data and provide reliable analysis. 

Dragin's underwriting tools are built specifically for alternative finance. Trained on thousands of financial documents, it understands the industry context, what information underwriters need, and what matters when evaluating a deal. Beyond identifying revenue, NSFs, or financing positions, Dragin can cross-check submitted files and online resources to detect red flags and risks, along with cited links and confidence score. 

Want to see how it works in practice? Book a demo with Dragin 

References 

Balsiger, D., Dimmler, H.-R., Egger-Horstmann, S., & Hanne, T. (2024). Assessing large language models used for extracting table information from annual financial reports. Computers, 13(10), 257. https://doi.org/10.3390/computers13100257 

Deußer, T., Leonhard, D., Hillebrand, L., Berger, A., Khaled, M., Dilmaghani, T., Kliem, B., Loitz, R., Bauckhage, C., & Sifa, R. (2023). Uncovering inconsistencies and contradictions in financial reports using large language models. 2023 IEEE International Conference on Big Data (BigData), 2814–2822. https://doi.org/10.1109/BigData59044.2023.10386673