Code Interpreter guide for product managers
How can Code Interpreter supercharge your discovery? What are the use cases relevant for PMs? How can you hack through existing limits and avoid data privacy issues? Read on.
Hi and welcome to the Corporate Waters weekly newsletter 🙌
I’m Mikhail and I'm excited to share my learnings to help you navigate the murky waters of product management, leadership, and corporate dynamics. Subscribe to unlock the full value.
In today’s FREE newsletter:
Limitations of Code interpreter;
How to integrate it into your workflow and avoid data privacy issues;
What are the specific use cases of Code interpreter for product managers;
Actionable tips & tricks to overcome existing system limitations.
There’s a lot of hype in the media about Code interpreter (CI) making Data analysts obsolete. However, those clickbait claims hold little substance. If you look past the sensationalism, there’s almost zero utility in most of the published content on the topic.
Our team had access to CI for a few months even before a public rollout. We’ve flexed it far and wide to figure out what works and what doesn’t. It’s not going to replace Data Analytics anytime soon, but it can dramatically reduce the time spent on ad-hoc requests from product managers.
How can CI help product managers to supercharge discovery? What use cases are relevant for product managers? How can you hack through the existing limits of CI? How can you integrate it into your existing workflow without risking privacy? Read on to learn more.
⚠️Limitations
There’s a range of constraints that CI has. Some of them are added by design, and some are actual flaws.
Design Limits
🚫 Token limit. There’s a maximum 8k symbol context limit for Code Interpreter. It’s sufficient for most ad hoc requests. If you want to do heavy stuff, however, there are workarounds.
🚫 Limited amount of libraries. There’s a preset Python environment with 330 most frequently used libs. You cannot install external libraries for safety reasons. This is a problem, because CI isn’t aware of this limitation and frequently attempts to access libraries outside of it’s scope. The result is a mistake, which it tries to correct, eating up your tokens.
🚫 GPU limits. Although there are no specific guidelines, you won’t be able to run computation-heavy ML/model training. An issue for Data Science, but no biggie for an average product manager.
🚫 File size limits. You can upload a file (*. txt, *.csv, *.json, *.xls, *.xml, images and video) up to 512mb. There’s something wrong with your dataset if it’s more than 512 MB. Anyways, you can always clean it up, split the file in chunks before uploading it to the Code Interpreter.
Flaws
🚫 Data Privacy. Because the input data you provide is used for model training + who knows whatever else, there’s currently no way to establish a safe and independent environment from OpenAI. Hence, you cannot fully integrate the Code Interpreter into your existing workflow and data lakes.
🚫 Hallucinations. The more ambiguous your prompt is, the more creative the Code Interpreter becomes. It can come up with non-existent libraries, variable names, and assumptions not grounded in the data you provide.
🚫 Looking for shortcuts. Unless you are very specific, CI will constantly look for shortcuts (just like a junior analyst does). Eventually, you can end up with variable names, graphs or recommendations that are not very clear and require brushing up.
🚫 Descriptive vs. analytical. The output skews to being more descriptive than analytical. It can give you a proper breakdown and description of each of the variables in the dataset, but fail to fully analyze the data and give you conclusions. The only workaround is to be very precise and explicit in your prompts.
Due to inability of full integration of Code Interpreter to your existing workflows, design limits and intrinsic flaws I don’t see how it can fully replace a Data Analytics function. On the other side, it can already become a proper analytics partner. As As OpenAI notes themselves that CI performs like “an eager junior data analyst”. I would say that it’s a data analyst with senior knowledge, but junior execution skills and unbeatable work ethics.
🔧 Setting up your workflow
Yes, you cannot connect CI to your existing data lake or openly share the data. What you can do, however, is employ some basic cryptography before publicly sharing the files. Once you’ve performed all the analysis you need, you can decipher the data back.
Encoding your data
Download a raw *.csv from your Tableau, Looker, or MySQL environment
Numbers: Change the actual numbers equally by whatever order of magnitude it makes sense - divide by 2, multiply by 1/166 e.t.c. (excluding the dates)
Letters: Cipher the text with the most simple “Caesar cipher” that reverses the letters.
You can do this locally by copy-pasting the following *.py script (which was generously generated by ChatGPT as well):
import csv
def process_csv(input_file, output_file):
with open(input_file, 'r') as infile, open(output_file, 'w', newline='') as outfile:
reader = csv.reader(infile)
writer = csv.writer(outfile)
for row in reader:
new_row = [str(float(item) / 4) if item.replace('.', '', 1).isdigit() and '/' not in item and '-' not in item else
''.join(chr((ord(c) - 65 + 13) % 26 + 65) if c.isupper() else chr((ord(c) - 97 + 13) % 26 + 97) if c.islower() else c for c in item) if item.isalpha() else
item for item in row]
writer.writerow(new_row)
input_file = 'input.csv' # Your input CSV file
output_file = 'output.csv' # The output CSV file
process_csv(input_file, output_file)Make sure you save the *.py file to the same folder as your *.csv dataset. If the code returns an error, provide an absolute path to your file:
input_file = '/Users/YourUserName/Documents/input.csv'Deciphering your data
Once CI has made the analysis you need, and you’ve gotten the *.csv file, you can reverse the transformations by using the following Python script:
import csv
def process_csv(input_file, output_file):
with open(input_file, 'r') as infile, open(output_file, 'w', newline='') as outfile:
reader = csv.reader(infile)
writer = csv.writer(outfile)
for row in reader:
new_row = [str(float(item) * 4) if item.replace('.', '', 1).isdigit() and '/' not in item and '-' not in item else
''.join(chr((ord(c) - 65 - 13) % 26 + 65) if c.isupper() else chr((ord(c) - 97 - 13) % 26 + 97) if c.islower() else c for c in item) if item.isalpha() else
item for item in row]
writer.writerow(new_row)
input_file = 'output.csv' # The modified CSV file
output_file = 'reversed.csv' # The CSV file after reversing the changes
process_csv(input_file, output_file)
💼 Use Cases
Essentially all the Data Analytics methods can be broken down into four major buckets that differ by the complexity and the value they can provide.
The further we look into the future, the more complex the analysis becomes, and the more critical reasoning is required. For discovery purposes, we don’t expect product managers to have extremely deep knowledge in all of the buckets. Usually, the simplest of the methods suffice: build a distribution, calculate averages, do some data discovery and root-cause analysis, and at the worst, do a linear regression.
Let’s see how well CI can perform in all of those buckets by testing the core use cases that a PM might be faced with.
📝 Descriptive
Data Cleaning
Sometimes the data you download is pretty messy. Plenty of variables you don’t need, missing values in certain fields. Previously, it would have taken me an hour or so to do a decent clean-up and have a dataset ready for work. With CI, you can do this in seconds.
Prompt: Clean up the data and just leave the columns with the names “X”, “Y” and “Z”
Build a distribution
One of the simplest ways to make your data readable is by building a distribution. CI nails it in under a minute. You have to be specific about the columns containing the required data; otherwise, CI can simply interrupt the analysis and explicitly ask for guidance.
Prompt: Build me an age distribution of squirrels (column "Age") and provide clear visuals
In the example above, I’m using a NY Squirrel Census dataset (all the squirrels noticed in the Central park in 2018). As we can see, the majority of squirrels are adults, ~15% are juveniles, and for the rest, the information is either missing or the age is uncertain.
Cohort and user segmenting
When you need to understand the profiles of your core power users, you might want to refer to CI’s user segmenting capabilities.
In the example above, we’re using a sample dataset of 512 Spotify users. CI goes into segmenting users by each of the provided criteria (age, gender, subscription plan, usage period, preferred content, preferred devices). Within two minutes, it gives you a decent overview of the core user base. You can spice up your request further with:
Prompt: Where does the biggest opportunity for business lie?
It will output 6 ways the business can improve (converting free users to premium, attracting older users, expanding podcast listenership, targeting male and other users, improving experience on other devices, increasing usage among new users). Of course, those are high-level recommendations, but they can already serve as a guideline for further research.
🔍 Diagnostic
Hypothesis generation
If you don’t know where to start your analysis, you can ask CI to do the hard work of thinking it through.
Prompt: Generate a draft hypothesis and test it with the data, make assumptions and provide compelling reasoning
In the provided example, I’m using a dataset of 130k wine reviews from Winemag over 2012-2017. CI came up with a reasonable hypothesis on the correlation between price and the wine rating. Even though the correlation turned out to be mildly positive 0.4, there were outliers in each price and rating category, meaning that there are other factors at play as well. At this point, since the token limit wasn’t fully utilized, you can push it forward to make a deeper analysis. You can even spice it up with:
Prompt: Come up with unexpected observations and provide a compelling reasoning
Heatmaps with recommendations
You can do a quick heatmap generation/grids based on your inputs.
Create a grid of crimes on the map with strong visuals. Provide compelling recommendations based on those inputs.
In the provided example, we’re using an LA crime rate dataset, which covers incidents from 2020 to 2023. Even though CI stumbled and self-corrected a few times (wrong variables, forgetting to use some of the libs), it has created a more or less readable heatmap with longitude along the X-axis and latitude along the Y-axis. The output is not very user-friendly, but still gives a preliminary understanding. Unfortunately, a proper geo-chart isn’t possible as CI doesn’t have access to the relevant libs, but you can ask for code snippets and execute it on your local machine.
Seasonality correlations
Is there a seasonality correlation in beer consumption? Let’s figure it out.
Prompt: Is there a seasonality correlation in beer consumption?
In the example above, I’m using a small dataset of recorded beer consumption in São Paulo. CI was able to identify the limitations of the dataset (not having two complete yearly cycles) to build a proper seasonality correlation. The result was inconclusive, but the methodology and the process were correct.
🔮 Predictive
Linear Regression
Let’s try to predict coffee prices in the upcoming 50 years based on historical data from the following dataset.
Prompt: Predict how the coffee prices would change in 50 years using linear regression
Even though CI stumbled multiple times (missing values, added spaces in variable names), it eventually self-corrected and gave out a 50-year prediction graph. On the other side, the graph isn’t very readable (CI is using days instead of years), hence you have to be very specific in your prompt inputs or ask it to self-correct afterward.
Clustering
If you want to group massive datasets into readable clusters, you can request CI to do the heavy-lifting for you.
Prompt: Cluster the reviews, come up with clear cluster names, provide compelling visuals. Only use the integrated libraries.
Since clustering involves multiple steps of data processing, feature extraction and grouping, CI can easily go sideways and refer to the missing libraries. The workaround is to add “Only use the integrated libraries”. It helps prevent ~70% of the mistakes and reduce the token consumption.
In the example above, we’re using a cleaned dataset of IMDB Barbie reviews. Through trial and error, CI estimates the maximum amount of clusters and interprets the clusters as “Reviews discussing the film as a whole”, “Reviews containing spoilers”, “Reviews focusing on personal opinion of the film”. Unfortunately, due to the context limit, the CI fails to make proper visuals of the clusters.
💊 Prescriptive
Stochastic Simulation
Usually we’re dealing with the world of complex probabilities, and simple regression isn’t enough. Can CI predict three scenarios of hotel booking demand that incorporate cancellations and length of stay? Absolutely.
Prompt: Build a stochastic simulation of hotel demand and group the outputs in three stakeholder friendly scenarios
In approx. 3 minutes, we’ve gotten an analysis that would typically take 1-2 weeks to accomplish. This was done purely for illustrative purposes, as rarely do product managers have to go that far.
Strategic Recommendations
Let’s flex our critical reasoning muscle and try to figure out what opportunities lie ahead for Uber based on the Q2 Earnings report.
Prompt: Read the Uber's Q2 Earnings report and provide a list of business opportunities with compelling reasoning.
Well, the output is a bit vanilla and doesn’t go beyond the obvious. It just lists the known facts and emphasizes the opportunities that open up in light of this information (e.g., record cash flow of €1.1B can allow the business to explore new acquisitions). CI cannot yet deal with such ambiguity, and you’ll have to be more specific by providing additional context.
🎩 Tips & Tricks
✅ Increase the Context
The way to overcome the 8k symbol limit of your CI session is to ask it to give you a dump *.csv file with the context that you can later re-upload and continue the session.
Prompt: After each analysis step give me a *.CSV file with all the context above
✅ Use Only Existing Libraries
Sometimes CI starts thinking creatively and goes outside of the scope of the integrated 330 libs. The way to overcome this is by providing the explicit instruction.
Prompt: [Your prompt]. Only use the integrated libraries.
✅ Avoid Simple Mistakes
Sometimes CI forgets to clean up the data, disrupting the analysis and consuming more tokens. There’s a simple way to overcome this.
Prompt: Clean the data and [your prompt].
✅ Increase the analysis speed
You can ask CI to be as fast as possible; hence, it will have to choose the most pragmatic and cost-efficient analysis methods.
Prompt: Accelerate the analysis [your prompt]
✅ Reduce Hallucinations
In order to avoid random variables, libs and data to pop up, you have to provide CI with explicit instructions.
Prompt: Only use the integrated libraries and the variables and data provided in the file.
✅ Spice It Up For Stakeholders
Sometimes CI comes with “not very user friendly” outputs and variable names. You can ask it to spice things up for you.
Prompt: Spice up the variable names so they are readable by stakeholders.
✅ Don’t Let It Give Up
In search for the shortcuts, CI sticks to what is high-level or descriptive instead of going really deep. You can brute-force this tendency with the following prompt:
Prompt: Go as deep as possible and do not stop the analysis until I tell you so
✅ Solve a Subset of the Problem
If the problem is too complex (e.g., Markov chain), you can break it apart and solve only for a subset of the problem.
Prompt: Create a transition matrix for the uploaded states based on the provided data inputs
🗝️ Key Takeaways
CI is impressive and can execute data analysis methods of high complexity (predictive, prescriptive). But as with every junior specialist, it is constantly seeking shortcuts and doesn’t always provide you with the pitch-perfect output. Hence, you have to guide it with context and very explicit instructions (just like you would manage a junior data analyst).
There are design and system limitations, but you can find a workaround almost for every single one (context limit, hallucinations, giving up, increasing the speed, spicing it up). Yes, you cannot fully integrate CI into your data lake and fetch the data freely, but you can execute proper ad-hoc data analysis. In order to avoid data privacy issues, you'll have to encode your data before sharing and decode it afterward.
Based on the list of examples ranging from descriptive to prescriptive analytics, product managers can save time by integrating CI into their existing workflow. If you're not doing it already, I hope this article will help you revisit your current stance.