Finally Found a Local LLM Setup That Seems to be Promising

 

 

AI Image 

The bill problem

I love Claude. I love ChatGPT. I use Gemini too. All three, for work. But at the end of the month, the bills arrive. Like a relative who comes for a day but doesn't want to leave.

For office work, its fine. But for my personal stuff? My open source projects, homelab setup, random late-night experiments. Paying for every token felt wrong. Especially when I have good machines with decent hardware.

So I wanted a local LLM. On my own machine. No meter running in the background.

I tried. Many times. I tried Cline, Openclaw, etc. I tried a bunch of other tools I don't even remember now. Every time, same story. Either the tool froze. Or the model told me things that were not true, with such confidence. I felt like talking to an uncle at a wedding about crypto. Honestly, I gave up. More than once.

Then came Cindy

I follow the AI Weekly newsletter. The last issue mentioned Cindy. An open-source AI agent that "works out of the box." I have heard "works out of the box" before so many times that I am no longer amazed to find the box empty now. So, I tried it anyway. And this time, something was different.

A little about Cindy. Its open source, Apache 2.0. It runs on your own machine and works with your real files. You can bring your own model, including a local one. That last part was all I needed to hear.

Then I read the tagline. "Consider it done." Wait. I have heard this before. If you did engineering in Pune, you have too. Syntel used "Consider IT Done" for years. Capital IT, of course. Besides, Syntel and Cindy sounds same too 😀. Cindy's tagline is "Think it. Consider it done". So I got it setup.

My setup

• Machine -> MacBook Pro, M3 Max, 48 GB unified memory
• Model runner -> Ollama
• Model -> Qwen3-Coder 30B
• Agent -> Cindy

Setup was fairly simple on Mac, Linux is a bit complex, I am going to do that next:

1. Install Ollama and pull the model: ollama pull qwen3-coder:30b
2. Download Cindy from cindy.app.
3. On the login screen, choose Skip Sign-In. This runs local agents. No Cindy account needed.
4. Point Cindy to the local Ollama model.

What did I use it for?

I have a homelab. A Kubernetes cluster. A monitoring stack. A few hobby apps. Earlier, I used Claude and ChatGPT to keep an eye on them. Now I asked Cindy simple questions:

• Which pods are not running? Did anything restart recently?
• Is my monitoring actually monitoring, or just sitting there looking pretty?
• It also did some administration jobs for my SSHing to baremetals and monitoring processes, setting up cron etc.
• Some internet research and documentation bit. Yet to try programming.

And here is the surprising part. It does not make things up. When a pod was down, it said the pod was down. When all was fine, it said all was fine. No imaginary CrashLoopBackOff for extra drama. I don't know the exact secret sauce. My guess? Cindy behaves like an agent. It checks real output instead of guessing from memory. Whatever it is, somebody engineered it well.

Let me not oversell. It is slow. Claude and ChatGPT reply in seconds. A local 30B model takes its time. Sometimes I make chai while it thinks. But I don't mind. For my homelab, speed is not the goal. Accuracy is. I would rather wait two minutes for a correct answer than get a wrong one in two seconds.

Next stop: Linux

Mac is sorted. Next, my Linux machine. Cindy has a proper Linux guide. Ubuntu 22.04 and 24.04 get an official .deb through apt. Arch-based systems get a one-line install, as your normal user. It can be a bit tricky. I am going to try this on Manjaro(Yes, I know Manjaro is not real Arch).

Feel free to try this and let me know how it works for you.

#AIAgents #Documentation #ContentCreation #DevOps #CreativeWork

✏️ drafted with ai assist, because practicing what I preach.

MeasureCamp Delhi: Where I Became a Speaker Without Meaning To

 

Last week I went to MeasureCamp Delhi 2026. This was their 200th event and indeed a special one. Before going, I had a simple picture in my head. Rows of chairs. A stage. Some speakers talking. Us nodding along. Chai/Coffee in between. Maybe a goodie bag at the end. People looking to sell their product every now and then. Group of students clicking selfies all over. The standard conference deal we have all attended at least fifty times in our careers.

I was wrong indeed.

MeasureCamp is something called an "unconference." When I heard the word for the first time, I thought maybe they forgot to plan the conference part. It was on purpose though. There is no fixed agenda. The event starts with a short intro, sponsors get their thank you, and then they show you a big blank board with time slots and room numbers. That board is the real hero of the day. Here is a picture:

Speaker/Event Session Board

I remember pre-covid days, I did something similar. As a meetup organizer at DevOps Pune and after getting some inspiration Seattle meetup groups, I started working on Leancoffee format of meetups since we were getting lot of marketing pitches back then. Here is what Leancoffee is. Definition copies as it is from their site.

Lean Coffee is a structured, but agenda-less meeting. Participants gather, build an agenda, and begin talking. Conversations are directed and productive because the agenda for the meeting was democratically generated.

So I now assumed this Unconference was probably the same, with a new name. But again, I was wrong.

Here is what happens. If you want to speak on something, you write it on a small card, stick it on the board, and boom, you are now a speaker. No approval committee. No forms. No "we will get back to you." If people like your topic, they walk in. If they don't, you get a very empty room and a good lesson in humility.

Only one rule was strict. No pitching your company. Sponsors already pay for that privilege, so the rest of us just talk shop, not sales. Once that rule was set, every conversation in the building felt honest. Nobody was secretly selling you a subscription while pretending to chat.

Wow! It sounded good and scary at the same time. Good, because

• This was different. A different learning experience is always good.
• No marketing pitches. As an organizer, I know how bad this can get.
• I could choose who and what I want to listen to.
• So many good topics being displayed and so much to learn.

Scary because:

• Embarrassment of not having any audience if you end up becoming a speaker.
• Fear of missing out on good speakers if multiple good events go in parallel.

My colleague Rohan Deshpande and I went there just to attend, representing Convert.com, nothing more. No plan to speak, no slides, nothing prepared. But after the opening session, we looked at the board, looked at each other, and thought, why not? We put up a session on how AI has been sneaking into troubleshooting and internal workflows at CRO and A/B testing companies. Basically the stuff we have been quietly doing for months.

We kept it in a very informal discussion format. In fact, I sat down with the attendees just facing them. More like a group chat than a lecture. About 15 to 20 people joined. And the questions came fast. Do you actually trust AI's calls or are you just being brave? Where is this going for testing platforms? Personalization, traffic tracking, integrations with tools people already use daily. People were genuinely curious, not just polite.

Best part of the whole day, honestly. We did not plan to speak, did not prepare a single slide, and still walked away with speaker mementos. Somehow that felt more earned than if we had rehearsed it for a week.

This time the turnout was around 120 plus people. I think they expected more, but Delhi was also hosting the BRICS summit that same day, so a lot of people probably got stuck in traffic or security barricades instead of analytics discussions. Still, seven breakout rooms ran side by side the whole day, and people moved between rooms freely, following their curiosity, not any fixed schedule.

I also sat in on a few sessions myself. One on attribution problems in ads and CRO. One on moving from GA4 to Amplitude. One on data governance and AI. And one with a title that stuck with me, "How Data Lie." Every session was run by someone actually doing the work, not someone presenting a fancy case study made for likes.

But if I am honest, no single session was the best part. The people were. I met folks from Amplitude, founders running their own analytics platforms, and CRO experts who have worked with some really big consumer brands. Normally this kind of networking takes weeks of cold emails and LinkedIn requests that nobody replies to. Here it happened in five minutes near the coffee counter.

This was also the 200th MeasureCamp Delhi event, which is no small number. A big thank you to the organizers and volunteers who pulled this off. Running an unconference sounds easy on paper, "just let people talk," but making that chaos actually work smoothly takes real effort. They did a solid job. Thank you so much for all the work. Tanmay Garg Priya Dhiman Anuj B. Abhishek Deswal Sorry if I missed anyone here.

If you work anywhere near analytics, data, or experimentation, and you have never been to a MeasureCamp, go once. Go with zero expectations. That is honestly the whole point.

Disney Did It With Cartoons. My AI Did It With My Kid's Bedtime Story.

 

I saw a video a while back about old Disney animation and how they fooled viewers in the last couple of years. Same character movements, reused frame by frame, across completely different characters and different movies. Different faces, different worlds, same underlying motion. It worked for years. Nobody noticed, or nobody cared, because the backgrounds and story kept changing enough to sell it. Here is the video I am talking about. This was deliberate for many reasons as you read more about it, you will know. Some reasons, I understood were, nostalgia, saving time or maybe laziness too.

I didn't think much of it until I saw the same thing happening with AI.

I was working on generating some images recently. I kept feeding new context, new prompts, expecting newer results. The first couple of images were just fine. Then I started noticing a pattern of same poses, same layouts, same compositions repeating underneath a different backdrop. Not identical, but close enough that it felt like the AI found something that worked and decided to stop being creative.

I told myself, okay, maybe that's fine for images. Then I tested it on something that mattered more.

I use AI sometimes to teach certain concepts to my kid. I asked AI to write a short story. It was about living and nonliving things. It came out lovely. She enjoyed it, I enjoyed it. All good, right?

The next day, I asked for a story about vertebrates and invertebrates. Same structure. Same pattern with just a bit of find and replace may be 😀. Practically the same story with a few words swapped out. If I had outsourced that to a person and gotten this back, I wouldn't have accepted it. And here's the thing, my kid is five. Even she would note this eventually. Kids get bored of the same story wearing a different costume faster than we think.

Now this is not about things impacting my personal work. The part that actually worries me is documentation.

I use AI heavily for documentation and writing. And I've caught the pattern there too. Same structure repeating article after article, same phrasing patterns, same rhythm, regardless of what the topic actually needs. It's  probably efficient. But it's also lazy. And I worry people will lean onto this instead of catching it, hand AI a rough context, ask for a full end to end document, and publish whatever comes back without questioning why every article somehow reads the same.

Here is what I found being the technical reason to be. This comes down to how models pick outputs. They default to the highest-probability path unless you push them elsewhere, so once a structure works well in one response, it becomes the easy option for the next, especially in the same session where earlier context keeps shaping what comes after. Lower temperature and conservative sampling, which most tools default to for reliability, make this worse on purpose. It's not laziness the way we mean it. it's optimization doing exactly its job: minimize risk, maximize consistency. Creativity was never the objective. Which is exactly why the paragraph-by-paragraph, keep-pushing-back approach works, it forces the model off the safe path it would otherwise default to.

When I see this happening, here's what works for me instead. Don't ask for the whole thing to be written or documented at once. Go paragraph by paragraph. Feed fresh context deliberately every time. Question, Argue and provide newer/better context every time. And at any time something feels repeated, say so. Tell it directly. That nudge matters more than people think, it's not just fixing one document, it's teaching the model what you don't want. Some would comment and ask to create a new session for improvement here. I agree with it, but how I and sometimes even you may work, sometimes we need the session memory to exist for various reasons, so we use the same sessions for a long time.

Disney may have eventually moved past reusing the same animation cycles. Or even if they re-used some, they had some valid reason like ‘nostalgia’. They could have even stopped doing this because audiences quietly noticed this and started expecting more, and also maybe that the reused motion stopped being good enough. I think we're also at the exact same point with AI generated content, whether that's documentation, images, or bedtime stories. It'll get away with repeating itself for a while. Until the audience notices. And audiences always notice eventually, even five year old ones will.

So if you're using AI for content, don't just accept the first draft's texture as final. Push it. Ask it to genuinely think differently, not just swap a few nouns. Creativity isn't AI's job to protect. It's ours.

#AIAgents #Documentation #ContentCreation #DevOps #CreativeWork

✏️ drafted with ai assist, because practicing what I preach.

Documentation Drifts. Here's How to Catch It.

 

"Documenting the documentation."

Yes, I know how that sounds. Documentation about documentation about documentation. Somewhere an intern is crying. Actually that's the phrase that hit me mid-conversation yesterday.

In DevOps we talk about monitoring the monitoring system. You don't just set up alerts and walk away. You watch the watcher too, because if your monitoring breaks silently, you find out the hard way. We handle it mostly by using a 3rd party monitoring Saas tool or a completely different monitoring tool that is on a different infra to monitor our monitoring. Many a times, we also setup multiple monitoring tools to monitor each other.

Turns out documentation has the exact same blind spot, that I realized yesterday.

Here's the problem. Documentation itself isn't static, but the system around it should be. A proper guideline of how the docs should look consistent across.

Think about what happens with two or three writers on the same knowledge base.

1. One person is deeply technical, every article turns into a dense reference doc.
2. Another writes for humans first, simple language, easy to follow, but a senior engineer might find it too light.
3. A third writes in a fun, engaging tone that people love reading but that doesn't actually answer the question they came for.

None of these people are wrong. But put them in the same wiki with no shared rules, and the documentation stops looking like one system. It may end up looking like three different people arguing.

That's what documenting the documentation fixes. Not the content. The structure around the content. Heading formats. Whether an index is required. How many bullets before it's too many. Screenshot rules, including the boring but important one, don't reveal internal or customer details in a screenshot(Nothing says 'security incident' like a screenshot where someone forgot to close the tab with the customer PI sitting right there). Whether emojis are fine or banned. Which kinds of diagrams are expected. Small decisions, but decided once, so nobody has to reinvent them article by article. Is humor ok. How much humor is ok? Procedure to build the context, etc.

Two things break without this:

1. When someone leaves the team, whoever replaces them has nothing to stand on. No rulebook, no shared style, just vibes and old examples to guess from.
2. Even with the rules written down, you can't assume people will keep following them. Rules you hand someone on day one quietly erode by month six. Rules handed over on day one have a shelf life shorter than office milk.

So the system needs an audit layer too. Not a one-time handoff. A recurring check.

This is where it gets easier than it used to be. You write your rules once, in something like a CLAUDE.md, plain and specific. Then you let AI do the boring, repetitive part; scan every new article, and periodically scan the old ones too, and flag anything drifting from the standard. Wrong heading structure, missing index, a screenshot that shouldn't be there, tone that's drifted too casual or too dense. The AI doesn't decide what's true. It checks what's consistent. Humans still review before anything changes.

Monitoring the monitoring system caught on in DevOps because someone finally said the obvious thing out loud, watch your watchers. Documentation needs its own version of that sentence.

Document the documentation. Then audit it like you mean it.

#Documentation #DevOps #KnowledgeManagement #AIAgents #TeamProcess

✏️ drafted with ai assist, because practicing what I preach.

Credential bleed: what happens when your MCP server trusts the wrong thing

 

I set up an MCP server for MySQL last month. Took maybe twenty minutes. FastMCP, running on port 9100, exposing a handful of database tools to any agent that connected to it.

Then I asked myself a question I should've asked before writing a single line of code. What exactly is this server trusting, and who gave it permission to trust that?

Turns out, the answer was: whatever connected to it.

Here's the thing about MCP servers that nobody warns you about when you're following a quickstart guide. The server doesn't inherently know the difference between "an agent I authorized to run read queries" and "an agent that happened to find my endpoint." If your auth model is weak or missing, every tool you expose is available to every connection that shows up. Credentials don't need to be stolen. They just need to be reachable.

I built this specific setup as a demo, deliberately, to see how bad it could actually get. Not theoretical. A real MySQL MCP server, real credential handling, real tools exposed. And what I found is that credential bleed isn't some exotic attack. its the default state of a lot of MCP setups people are shipping right now, without realizing it.

Think about what an MCP server actually does. It sits between an agent and a real system, translating natural language intent into real actions. Real database queries. Real Terraform applies. Real file writes. Somewhere in that translation layer, credentials have to live. And if that layer trusts the connection instead of the identity, you don't have an access control system. You have an open door with a "please don't" sign on it.

This is the same failure mode I saw with a Terraform agent demo I built separately. An ambiguous prompt walked straight through terraform_plan into terraform_apply and created a real S3 bucket in AWS. Nobody meant for that to happen. The agent didn't misbehave. The boundary that was supposed to stop it simply wasn't there. Same root cause, different door.

The uncomfortable part is that this scales with adoption, not against it. The more useful MCP servers get, the more tools get exposed, the more valuable the thing sitting behind that unguarded connection becomes. A vulnerable server nobody uses is a curiosity. A vulnerable server your whole team routes agent traffic through is a real incident waiting on a slow Tuesday.

None of this means don't build MCP servers. It means stop treating the connection as the identity. An agent proving it can reach your server is not the same as an agent proving it should be trusted with what's behind it. Ephemeral credentials, scoped tools, intent validation before execution, these aren't nice-to-haves you add later. They're the actual product. Everything else is just a demo that works until it doesn't.

I'll probably write up the full technical breakdown of the MySQL demo separately. For now, if you're standing up an MCP server this week, ask the boring question first. Not "does it work." Trust what, exactly.

Part 3: How to Decide Which CRO Test to Run First

Because the loudest idea in the room is not always the best experiment.

In the last post, I wrote about the main experiment types: A/A, A/B, Split URL, and multivariate testing. I also introduced the formula that quietly changed how I think about experimentation:

Success = Chance × Frequency

Once that clicked, I assumed the next step would be simple.

Pick an idea > Run a test > Measure it > Repeat.

Then I actually tried to do it. Because once people around you start thinking in CRO terms, ideas come from everywhere.

The designer wants to test the hero image. The founder wants to rewrite the pricing page because “it does not feel premium enough.” Someone from sales heard that a competitor changed their CTA and now wants to copy it immediately. Someone read a blog at midnight and suddenly wants to add a chatbot, a quiz, a floating banner, and possibly a dancing mascot.

Before you know it, your experiment backlog looks like a Jira board after a chaotic sprint planning session.

• Lots of tickets.
• Very little structure.
• Everyone convinced their idea is the one that will change the company’s destiny.

This is where prioritization comes in. Not because prioritization makes you smarter, but because it protects the experimentation program from becoming purely opinion-driven. And in CRO, opinions can multiply faster than credit card debts.

The Hidden Cost of Running the Wrong Test

At first, “let’s just test this” sounds harmless. CRO is about experimentation, right? Just run everything. But every test has a cost.

Even when the testing tool makes setup easy, a test still consumes traffic, time, design effort, analysis attention, and usually a couple of weeks of your roadmap. If your site has enough traffic to run two meaningful tests this month, and your backlog has twenty ideas, running the wrong two means you delayed the better ones. That is not a small cost.

This felt immediately familiar from Devops work. You do not deploy every feature request just because someone suggested it in a meeting with confidence and a nice slide deck. You triage. You look at impact, risk, effort, dependencies, and what the system is actually telling you before you start making changes.

CRO needs the same discipline. An idea is not valuable because it sounds smart in a meeting. It becomes valuable when it is connected to evidence and scored against other ideas in the backlog.

The HiPPO Problem

Before getting into frameworks, there is a force you need to understand. It quietly kills many CRO programs. It is called the HiPPO: the Highest Paid Person’s Opinion. Every team has one. In fact there is a rude saying about opinions. “Opinions are like…” (well… Google it)

The senior leader who says, “I think the button should be green,” and suddenly the next sprint is about button colors. The founder who saw a competitor’s landing page and wants to rebuild yours to match it by Thursday. The VP who has “a feeling” about the checkout flow. The stakeholder who says, “Can we just test it?” in the same tone people use before creating six months of technical debt.

To be fair, these inputs are not useless. People close to the business often have genuine intuition worth exploring. Sales teams hear objections. Support teams hear confusion. Founders understand positioning. Product teams know what users struggle with. The problem is not the idea. The problem is when the idea skips the queue without being scored.

In DevOps, we do not prioritize incident response based on who is shouting loudest in Slack. At least, we should not. We prioritize based on system impact and what the metrics are actually saying.

CRO needs the same structure. A prioritization framework is your polite defense against the HiPPO.

You are not saying: “No, your idea is bad.”

You are saying: “Great, let’s score it the same way we score everything else. If it ranks high, it goes first.”

That small shift changes the conversation. The framework depersonalizes the argument. The number does the heavy lifting. Very convenient, because numbers do not get awkward in meetings.

Start With the Funnel, Not the Idea

Before scoring individual test ideas, figure out where to look first. Not every page deserves equal attention. A page with very little traffic will take forever to produce statistically meaningful results. A page with high traffic but no meaningful business impact may still not be worth optimizing ahead of a page that sits directly in the conversion path.

Press enter or click to view image in full size

So before asking: “What should we change?” ask, “Where are we losing people?”, and then. Map your funnel.

For an e-commerce site, it might look like this:

Landing page > Product page > Cart > Checkout > Purchase

For SaaS, it might look like this:

Homepage > Pricing page > Signup > Activation > Paid conversion

For lead generation, it might look like this:

Landing page > Form view > Form start > Form submit > Qualified lead

Then look at where the biggest drops are happening. If 70% of users are abandoning at checkout, testing a homepage headline is probably not the first move.

• It might be interesting.
• It might even win.
• But it is probably not where the biggest leak is.

Analogy in DevOps: If the database is causing 90% of your latency, shaving 20ms off a frontend asset is not the priority. Sure, the frontend improvement may look nice in a performance report. But the database is still sitting there like an overloaded elephant in the server room.

Find the bottleneck first. Then fix it. In CRO, high-traffic pages close to revenue, signup, checkout, or lead generation are your hot paths. Start there.

Small disclaimer: Read at your own risk. This may make you hungry

The Two Frameworks: ICE and PIE

Once you know which pages to focus on, you need a way to score the ideas sitting in your backlog. Two common prioritization frameworks in CRO are ICE and PIE. Both are simple, both use overlapping dimensions and both are easier to remember if you stop thinking about frameworks and start thinking about food. Which, honestly, improves most business concepts. ;)

ICE: Ordering Food at Midnight

ICE stands for:

• Impact
• Confidence
• Ease

Think of ICE as ordering food on delivery app at 11 pm. You are hungry. You are tired. You are already in bed. This is not the time for adventure. You are not trying to discover a hidden culinary gem. You are trying to avoid sleeping angry. You see a biryani place.

Before you tap “Order,” three thoughts happen automatically.

First:

“How good is this actually going to be?”

If it is bad biryani, why bother?

That is Impact.

Second:

“Can I trust this place right now?”

Will it arrive hot, on time, and with raita? Or will it arrive cold, late, and missing the one thing that emotionally holds the whole meal together? You check the rating, reviews, delivery time, and whether someone recently complained that they received only rice and sadness.

That is Confidence.

Third:

“How easy is this?”

Is the restaurant open? Is delivery available? Will it reach in 25 minutes, or will the app say “arriving soon” until you lose hope? Will it ask you to verify your phone number again even though it has known you for five years?

That is Ease.

ICE is practical and slightly paranoid. It wants evidence before committing.

In CRO terms:

Impact asks: If this test wins, how much could it move the business?
Confidence asks: How sure are we that this is a real problem worth testing?
Ease asks: How difficult will this be to build, QA, launch, and analyze?

Score each dimension from 1 to 5. Then average the three. The highest-scoring ideas usually go first. ICE works especially well when you already have research data: heatmaps, recordings, analytics, surveys, support tickets, or past experiment learnings. Low confidence should lower the score, no matter how exciting the idea sounds.

Because “I saw this on a competitor site” is not research. It is browsing with ambition.

PIE: Choosing What to Order When You Have Options

PIE stands for:

• Potential
• Importance
• Ease

Now stay in the same food delivery app. You are still hungry. It is still 11 pm. You still should have eaten earlier like a responsible adult, but here we are. This time, you are not evaluating one biryani place. You are deciding what category of food deserves your attention first.

Biryani? Pizza? Rolls? South Indian? Chinese? That one “healthy bowl” option you add to cart and then remove after seeing the price? PIE helps you decide where the biggest opportunity is.

First:

“How good could this get?”

Could this be the kind of meal that fixes your mood, your evening, and possibly your belief in humanity?

That is Potential.

Second:

“How important is this meal?”

Is this a casual snack, or have you skipped dinner and now your stomach is sending production-level alerts?

That is Importance.

Third:

“How easy is it to get?”

Is it nearby and deliverable in 25 minutes, or is the restaurant far away, closing soon, and likely to cancel after making you wait?

That is Ease.

PIE is more optimistic than ICE.

ICE asks, “Can I trust this specific choice?”

PIE asks, “Where is the biggest opportunity?”

In CRO terms:

Potential asks: How much improvement could this page or funnel step realistically have?
Importance asks: How valuable is this page or funnel step to the business?
Ease asks: How simple or difficult is the test to implement?

PIE is useful when you are comparing broader areas of opportunity.

For example, if you are deciding whether to focus on the homepage, pricing page, checkout, signup form, or onboarding flow, PIE can help you decide where the biggest opportunity may be.

So, in food delivery terms:

ICE helps you decide whether to trust one restaurant.
PIE helps you decide which food category is worth exploring first.

ICE vs PIE: The One-Line Difference

Press enter or click to view image in full size

Here is the easiest way I remember them:

1. ICE helps you decide whether to trust one restaurant. PIE helps you decide which food category is worth exploring first.
2. ICE is about confidence. Can I trust this specific option to deliver? PIE is about opportunity. Which area has the biggest upside if I focus there? Same food app. Different thinking.
3. With ICE, you are asking: “Should I order from this biryani place?” With PIE, you are asking: “Should I even be looking at biryani first, or is pizza, rolls, South Indian, or Chinese the better opportunity tonight?”
4. Both end with Ease, because nobody wants unnecessary complications at 11 pm. Especially not when they are hungry and the app is already saying, “Restaurant is closing soon,” like it is adding pressure to your life.

Which Framework Should You Use?

Use ICE when your team already has research data and wants to prioritize based on evidence. ICE is useful when you are evaluating a specific hypothesis. For example:

“Should we move the CTA above the fold on the pricing page?”

You already have heatmap data. You know users are not scrolling far enough. You want to know whether this specific idea deserves to be tested next. That is ICE.

Use PIE when you are earlier in the process and still deciding where the biggest opportunity is.

For example:

“Should we focus on the homepage, pricing page, checkout flow, signup form, or onboarding?”

That is PIE.

At Convert.com, the Compass feature includes both PIE and ICE inside the hypothesis builder. When you create a new hypothesis, you can choose a framework, score each dimension on a scale of 1 to 5, and use that score to sort your backlog.

That helps because the score becomes visible to everyone. It is much harder to push your pet idea to the top when the number is sitting right there on the screen, judging you quietly. A visible score is like the delivery rating on a food app. You can still order from the suspicious restaurant. But now everyone can see that it has 2.3 stars and seven reviews mentioning “never again.”

Turn Ideas Into Hypotheses Before You Score Them

Press enter or click to view image in full size

One thing both frameworks require is that your ideas are written properly before scoring.

“Change the button color” is not a hypothesis.

“Move the CTA above the fold” is not a hypothesis.

“Make the page look more premium” is definitely not a hypothesis.

That is a mood.

A proper hypothesis looks like this:

If I change X, then Y will happen, because Z.

The “because” is the most important part. Without it, you are not testing an assumption. You are just making a change and hoping the graph goes up, which is basically the CRO version of deploying to production and whispering, “Please work.”

Or, to stay with the food delivery analogy, it is like ordering from a restaurant with no ratings because the photo looked nice. Could it be amazing? Maybe. Could it be regret in a plastic container? Also maybe.

For example:

If we move the main CTA above the fold on the pricing page, more visitors will start the demo request flow, because heatmaps show most users are not scrolling far enough to see the current CTA.

Now you have something worth scoring.

You know:

• The page
• The change
• The expected outcome
• The evidence behind it

Even if the test loses, you learn something specific. A backlog full of vague ideas becomes a dumping ground. A backlog full of hypotheses becomes a learning system.

Or in food terms:

“Order something nice” is not useful.

“Order from a highly rated nearby restaurant because delivery time is low and reviews mention fresh food” is much better.

Specificity saves you from bad decisions. And occasionally, bad paneer.

The One Thing ICE and PIE Get Wrong

ICE and PIE are useful, but they share a blind spot. They often treat every idea as if it exists in a vacuum. But it does not.

Every idea exists inside the history of your previous tests. If you have run five tests on your pricing page and four of them lost, your confidence score for the next pricing-page idea should reflect that history. Something on that page may be resisting your assumptions.

• Maybe the traffic quality is different from what you expected.
• Maybe the real problem is upstream in the funnel.
• Maybe people already understand the pricing page, and the actual friction is in signup.
• Maybe your pricing page is innocent and has been framed by bad hypotheses.

The same applies to food delivery. If you have ordered from the same restaurant four times and three times they forgot the raita, you should not treat the fifth order like a fresh mystery.

You have history. Use it. Past experiment results should feed into how you score new ideas. They should not sit inside a spreadsheet named something like: final_results_v3_really_final_updated_NEW.xlsx

This is why documentation matters as much as the framework itself.

After every experiment, write down:

• The hypothesis
• The result
• The confidence level
• The primary metric
• The key learning
• What it suggests about the next test

Even a losing test is useful if it tells you where not to look. Without this history, your backlog resets to zero every quarter. With it, your scoring gets more accurate over time because you are building a picture of what your specific audience actually responds to. A CRO backlog without documentation is like a food app that forgets every bad order you ever had. You keep making the same mistake. And somehow, the raita is still missing.

What to Do When Everything Scores the Same

Sometimes three ideas all score 4.2 and suddenly you are back in the same argument. This is normal. Frameworks help, but they do not eliminate judgment. A few tiebreakers can help. Pick the page with the most traffic. More visitors usually means faster results and less time waiting for statistical significance.

In food delivery terms, this is like choosing the restaurant with enough recent orders and reviews. If only two people have ordered from it since 2021, the rating may not tell you much.

Pick the test that is fastest to build. A high-scoring idea that takes three weeks of development work may be worth doing after a slightly lower-scoring idea you can ship in a day. Remember the frequency side of the success formula.

This is the difference between ordering something that arrives in 25 minutes and choosing a dish that says “preparation time: 95 minutes.”

Technically, it may be great. Emotionally, you may not survive. Pick the test that teaches you the most regardless of outcome. Learning value is underrated. A test that answers a fundamental question about your audience is worth more than a test that optimizes something already working reasonably well.

In other words, do not only ask:

“What might win?”

Also ask:

“What will we learn even if this loses?”

That question saves a lot of time. And occasionally, a lot of ego.

Define Success Before You Launch, Not After

One mistake that shows up in beginner backlogs is choosing the primary metric after the test has already started. That is dangerous. If you look at enough metrics, something will always appear to have improved.

• Clicks went up.
• Scroll depth changed.
• Form starts increased.
• Time on page looked different.
• Someone in Nebraska spent 14 minutes reading the footer.

But what was the experiment actually supposed to improve? Every test needs a primary metric before launch.

• Demo requests.
• Purchases.
• Trial signups.
• Form completions.
• Revenue per visitor.

Pick one. Commit to it. Let that metric make the call. This is like deciding what “good food” means before ordering. Are you optimizing for taste? Delivery speed? Price? Portion size? Not waking up with regret? If you decide after the food arrives, you can justify anything.

“The biryani was cold, but the container was sturdy.”

That is not success. That is coping. You can still watch secondary metrics for learning, but the decision should rest on the primary one. And alongside your primary metric, define your guardrail metrics. These are the things you are not directly trying to improve, but cannot afford to break. For example:

• If you are optimizing demo signups, lead quality should not drop.
• If you are optimizing checkout completion, average order value should not fall.
• If you are optimizing form completion, spam submissions should not explode.
• If you are optimizing clicks, downstream conversions should not suffer.

This is the CRO version of SLOs. You define what success looks like and what failure looks like before you are emotionally invested in the outcome. A test that increases clicks while damaging qualified leads is not a win. It is noise wearing a party hat.

Or in food delivery terms: A restaurant that delivers in 12 minutes but sends the wrong order did not “win on speed.” It failed dinner.

A Practical First Backlog

If you are starting from zero, here is the workflow I would follow.

• Map your funnel and identify the biggest drop-off point.
• Use heatmaps, session recordings, analytics, surveys, and any sales or support feedback you can access.
• Write down every test idea without judging too early.
• Convert each idea into a proper hypothesis using the “If X, then Y, because Z” format.
• Score each one using ICE or PIE.
• Pick one high-scoring, reasonable-effort test.
• Define the primary metric and guardrail metrics before launch.
• Run the test.
• Document the result and the learning.
• Then re-score the backlog based on what you now know.

That last step is what most teams skip. And it is what separates a backlog that compounds from one that stagnates. A CRO backlog should not be a parking lot where ideas go to quietly disappear. It should be more like a living system.

Or, to stay on brand, a well-maintained CI/CD pipeline for learning. Every test should make the next decision smarter. Just like every food order teaches you something.

• Restaurant is reliable.
• Delivery is slow on weekends.
• The “extra spicy” option is not a personality test you need to pass.
• Never trust a place where every menu item has the same stock photo.

Learning compounds when you write it down.

The Real Takeaway

Prioritization is not about finding the cleverest idea. It is not about the biggest redesign. It is not about the most senior opinion in the room. It is about running the tests that have the best combination of meaningful impact, strong evidence, reasonable effort, and clear learning potential. Done consistently, this turns CRO from a collection of random experiments into a learning system. You find the biggest leaks, collect evidence, write hypotheses, score ideas, run the most valuable tests first, document what happened and then repeat.

The DevOps parallel is hard to miss. Good infrastructure teams do not just ship more changes. They improve how they decide what to change, how they measure it, and how they learn from it.

Good CRO teams do exactly the same thing. And the food delivery parallel is also hard to miss. The goal is not to order everything on the menu. The goal is to make better choices each time, based on what you know, what matters, what is realistic, and what you learned last time.

That is prioritization.

In the next post, I want to go deeper into reading results without fooling yourself: statistical significance, confirmation bias, and why the most dangerous moment in CRO is when a test is almost winning.

I work with Convert.com and write about the intersection of DevOps and experimentation. If you are building your first CRO backlog or trying to move your team away from opinion-driven testing, drop a comment or reach out. Happy to compare notes.