ChatGPT for Data Analysis: Catch the Wrong Numbers

You type a question into an AI tool and seconds later you have a SQL query, a chart and a tidy paragraph about what it means. It feels done. Then your manager asks why your revenue number is higher than finance's, and nobody can explain it. That's the real risk of using ChatGPT for data analysis: AI-written analysis usually runs without an error and still gives the wrong number.

This guide is for analysts who want the speed without the embarrassing meeting. You'll see what AI handles well in SQL, spreadsheets and Python, the silent errors it makes, a worked query you can debug yourself, and a review routine that catches most problems before anything reaches a stakeholder.

What ChatGPT for data analysis does well#

AI is genuinely fast at the drafting layer of analysis:

These features are already inside the tools you use. Microsoft's Copilot for Power BI documentation lists creating report pages, summarizing reports and writing DAX queries.

What AI can't see is the context. It sees columns. It doesn't see the meeting where sales and finance agreed what "revenue" means, or the thread about the test accounts nobody removed. That context is your edge.

Where AI analysis goes wrong (and users have noticed)#

A Microsoft Research study of 200,000 Bing Copilot conversations found that among common work activities, the ones with the lowest share of positive feedback included "five visual design and five data analysis" activities (Working with AI, arXiv). People ask AI for data analysis a lot, and they rate the help lower than for other work.

The tool makers say it too. Microsoft's Copilot for Power BI page warns that without a prepared data model, "Copilot can misinterpret the data and return generic or inaccurate results." It also notes that when a question isn't about the data in your model, Copilot answers from the language model's general knowledge. That answer can look exactly like an answer from your data.

The errors to expect:

None of these throw an error. That's what makes them dangerous.

Worked example: debug an AI-written query#

You ask an AI tool: "Monthly revenue for Q3 2026 from our orders and payments tables." You give it the table names. It returns:

SELECT DATE_TRUNC('month', o.created_at) AS month,
       SUM(p.amount) AS revenue
FROM orders o
JOIN payments p ON p.order_id = o.order_id
WHERE o.created_at BETWEEN '2026-07-01' AND '2026-09-30'
GROUP BY 1
ORDER BY 1;

It runs. Three tidy rows. What to check before you trust it:

1. Fan-out on payments. In many systems one order can have several payment rows: a failed attempt, a retry, a partial payment. Run SELECT COUNT(*), COUNT(DISTINCT order_id) FROM payments WHERE order_id IN (...) for the quarter. If the counts differ, the join repeats orders, and failed attempts may be summed in.
2. Payment status. There's no filter on payment status, so failed or refunded payments count as revenue. Check which statuses exist and agree which ones count.
3. The last day. If created_at is a timestamp, BETWEEN '2026-07-01' AND '2026-09-30' stops at midnight at the start of 30 September. Orders from the rest of that day are missing. Check MAX(o.created_at) in your filtered rows.
4. Order status and test accounts. Cancelled orders and internal test accounts are still in. Ask whoever owns the definition of revenue what should be excluded.
5. One trusted total. Compare July's figure with finance's July number. If they differ, work out why before you share anything.

A safer version, assuming your team agrees on these definitions:

SELECT DATE_TRUNC('month', o.created_at) AS month,
       SUM(p.amount) AS revenue
FROM orders o
JOIN (
  SELECT order_id, SUM(amount) AS amount
  FROM payments
  WHERE status = 'succeeded'
  GROUP BY order_id
) p ON p.order_id = o.order_id
WHERE o.created_at >= '2026-07-01'
  AND o.created_at <  '2026-10-01'
  AND o.status NOT IN ('cancelled', 'test')
GROUP BY 1
ORDER BY 1;

Payments are totalled per order before the join, so each order appears once. Only successful payments count. The date range uses >= and < so the whole of 30 September is in. The status names here are examples; use the ones your database actually has.

Notice that asking the AI to explain its first query wouldn't have caught any of this. It would describe the query confidently, bugs included. An explanation is a second opinion, never a replacement for your row counts.

A prompt that shows you where the AI guessed#

Write a [SQL dialect] query that answers: [question].
Tables: [names, key columns, and how they join].
Definitions: [what "revenue" or "active customer" means here].
Rules: [date range, time zone, exclude test accounts or refunds].
Return: the query, every assumption you made, and a second query that
counts rows so I can check for duplicates.

The assumptions list is the most useful part of the answer. It shows you exactly where the model filled a gap.

Check the AI's summary too#

Say your result table shows:

And the AI writes: "North had the highest revenue per order, showing strong, reliable demand."

Do the maths. North and South are both $110 per order. West is $70. So "highest" is wrong (it's a tie), and "strong, reliable demand" rests on 15 orders, which is too few to call anything reliable. Check every sentence against the numbers, and soften claims built on small counts.

The review routine for any AI-written query, formula or script#

1. Read it before you run it. You can't check what you haven't read.
2. Check every table and column name against the schema.
3. Count rows before and after each join or merge. This catches fan-out, the most common silent error.
4. Check the edges: first and last date, NULLs, time zone.
5. Compare one total with a number you already trust, like a finance report or a count from the source table.
6. Check every sentence of the AI summary against the numbers.

Save your checks next to each query. In six months, that file is proof you work carefully.

Spreadsheets: test every AI formula on rows you can check by hand#

Formulas are one of the safest things to hand to AI, as long as you test them. Say you ask Copilot in Excel or Gemini in Sheets: "Sum the sales in column D where the region in column B is West and the date in column A is in September 2026."

You get something like =SUMIFS(D:D, B:B, "West", A:A, ">=2026-09-01", A:A, "<=2026-09-30"). Before you copy it down a report, check three things:

• Pick three rows you can add up yourself, including one on 30 September, and filter the sheet to confirm the formula gives the same answer.
• Look at the date logic. If column A holds date and time, "<=2026-09-30" can drop anything after midnight on the 30th, the same edge as in the SQL example. "<2026-10-01" is safer.
• Check for text that looks like a number or a date. Imported data often stores "West " with a trailing space, or dates as text, and the formula silently skips those rows.

Three rows and two minutes. That's the whole habit.

Is AI changing data analyst jobs?#

The studies point in different directions, which is worth knowing before anyone tells you a single number.

• Theory vs practice. Anthropic's Labor market impacts of AI found AI could perform 94% of tasks in computer and math jobs in theory, but covers about a third of them in practice today. Data analyst isn't in its top-ten exposed list.
• Some unemployment rise, cause unclear. A St. Louis Fed analysis found computer and mathematical occupations "saw some of the steepest unemployment rises" from 2022 to 2025, and the authors stress: "The findings presented here represent correlation, not causation" (St. Louis Fed).
• Employers expect growth. In the World Economic Forum's Future of Jobs Report 2025, "Data Analysts and Scientists" are among the fastest-growing jobs for 2025 to 2030.

Analysts online put it bluntly: if your job is only cleaning basic data and producing basic reports, AI absorbs a lot of it. If your job is answering real business questions with numbers people can trust, it doesn't. For the full picture on job risk, see our post Will AI replace data analysts?.

Skills that keep analysts valuable#

How to talk about AI in analyst interviews#

Weak: "I use AI for everything, it makes me ten times faster."

Strong: "I use AI to draft SQL and pandas code, then I check row counts and compare totals with a trusted report before I share anything. On my last project that caught a join that doubled revenue."

The strong version names a tool skill, a check and a result, and makes no claim about the future.

Free tools for data analysts#

• Project Brainstorm: pick a portfolio project with a public dataset
• Portfolio Writer: write up the project, including the checks you ran
• Free ATS checker: check your resume for "SQL", "Power BI" or "data quality", where true
• Skill gap analyzer: compare your skills with a senior analyst or analytics engineer post

Browse open roles: remote data jobs, data jobs in the United States and data jobs in India.

FAQ#

Can ChatGPT do data analysis? It can draft queries, formulas, scripts and charts quickly. It can't know your business definitions or tell whether the number is true. That checking is the analyst's job.

Is it safe to paste company data into ChatGPT? Only within your company's approved tools and rules. For practice, use public or made-up data, and paste schemas and column names rather than real customer rows.

What's the most common error in AI-written SQL? Join fan-out: joining to a table with several rows per record and summing, which inflates totals without any error message. Count rows before and after every join.

Should data analysts still learn SQL? Yes. You can't check a query you can't read, and checking is the part of the job AI doesn't do.

How do I prove AI skills as an analyst? A public project with the query, your checks and a short summary. Show what you caught, not just what you produced.

We're building short, hands-on courses on using AI in data analysis. Get an email when they open.