Parse or Perish: How We Extract Structured JSON from Unreliable LLM Outputs

You tell an LLM to return JSON. Sometimes it does. Sometimes it wraps it in markdown fences. Sometimes it adds a preamble. Sometimes it ignores the instruction entirely and returns a bulleted list. Here's the three-tier parsing system that catches all of it.

The Fundamental Tension

Replya's core API endpoint asks Gemini to generate exactly five LinkedIn comment suggestions and return them as a JSON array: [“Comment one”, “Comment two”, “Comment three”]. The prompt ends with the unambiguous instruction: “Return ONLY a valid JSON array. No explanation, no markdown fences, no preamble.”

And yet, in production, the model occasionally does every single thing we told it not to do. It wraps the JSON array in triple-backtick markdown fences. It opens with “Here are your suggestions:” It returns numbered lists instead of arrays. It adds trailing commas that break JSON.parse.

This isn't a Gemini-specific problem. Every LLM does this. The model's training corpus is full of examples where JSON output is presented inside explanatory text. The model is pattern-matching against a world where JSON lives inside markdown, not a world where it's a raw API response. You can fight this tendency with increasingly aggressive prompts, or you can build a parser that doesn't care. We chose the latter.

Tier 1: Direct JSON.parse

The first tier is the ideal path. We attempt a straightforward JSON.parse() on the raw content of the model's response:

In practice, this works about 80-85% of the time. When it fails, it's almost always because the response contains non-JSON text surrounding a perfectly valid JSON array. The model technically followed the instruction — the JSON is in there — but it wrapped it in something extra.

We could try to fix this at the prompt level. Add stronger language, use fewer tokens, structure the system instruction differently. But the diminishing returns kick in fast. Each additional line of prompt we add to enforce JSON formatting is a line we can't use to improve comment quality. The engineering trade-off is clear: accept 15-20% prompt non-compliance and handle it in the parser.

Tier 2: Regex JSON Extraction

When the direct parse fails, we move to regex. The insight is simple: a JSON array has a recognizable shape — it starts with [, contains some content, and ends with ]. We can extract it from surrounding text:

The [\s\S]* pattern is the workhorse here — it matches any character including newlines, which is essential for multi-line JSON arrays containing full sentences. The greedy match finds the outermost [ to ] pair, which is exactly what we want for a JSON array.

This catches another 10-15% of responses — the ones wrapped in markdown fences, preceded by explanatory text, or followed by a sign-off. On a busy day with hundreds of requests, that's dozens of users who see their suggestions instead of an error.

Tier 3: Line-by-Line Parsing

Sometimes the model returns a format so far from JSON that even regex extraction fails. Bulleted lists, numbered lists, paragraphs separated by blank lines. The content is there — the suggestions exist — they're just not structured as an array. Our fallback handles this with a line-by-line approach:

This approach strips common list markers — hyphens, bullet points, asterisks, numbers with dots or parentheses — along with any surrounding quotes. It filters out empty lines and takes the first N results. It's not elegant, but it's remarkably effective.

Critically, this tier catches maybe 2-5% of requests. Those are the edge cases where the model goes completely off-script — maybe it returned a numbered list with commentary between each item, or it generated markdown headings with the suggestions underneath. Without this fallback, those users would see an error. With it, they get suggestions that are usually just as usable as the ones from Tier 1.

The Post-Processing Step Nobody Talks About

There's one more step that runs regardless of which tier succeeded: we strip em-dashes. Every suggestion that makes it through the parser gets run through .replace(/—/g, ' ').

This seems almost too trivial to mention, but em-dashes are the single most reliable signal of AI-generated text. They're grammatically correct but socially wrong — real people on LinkedIn don't type em-dashes. They use hyphens, ellipses, or just start a new sentence. By stripping them at the parser level rather than trying to prevent them in the prompt, we eliminate the most visible AI tell without consuming prompt tokens on it.

Why This Architecture Matters

The three-tier approach solves a deeper problem than just parsing strings. It decouples output quality from output format.

In most LLM applications, these two concerns are entangled in the prompt. Every token you spend on “return valid JSON” is a token you don't spend on “write a good comment.” By handling format robustness in the parser instead of the prompt, we freed up prompt real estate for quality constraints — the persona design, the BAN list, the archetype variation, all the things that actually make the comments good.

This separation of concerns also makes the system more resilient to model changes. If we switch from Gemini 2.5 Flash to a different model tomorrow, the format quirks will change — but the parser doesn't care. It handles JSON, JSON-in-markdown, and unstructured text equally well. The prompt can focus on what it's best at (quality) and the parser can handle what it's best at (reliability).

Production Numbers

After thousands of requests, here's the rough breakdown of which tier catches which responses:

The key number is the 1% complete failure rate. Without the fallback tiers, that would be closer to 18%. That's the difference between a product that works most of the time and a product that works essentially always.

The Rule of Three

If you're building a product that depends on structured output from an LLM, here's the principle: never trust the format, always trust the content.

Your parser should have at least one fallback beyond the ideal path. The exact strategies depend on your output format, but the pattern is universal: try the clean parse, try recovery from common wrapping patterns, try a degraded but usable extraction. Three tiers. No less.

One tier is hoping the model behaves. Two tiers is being practical. Three tiers is being production-ready. Every additional tier you add after the first costs very little and saves you from the 3 AM page when a model update subtly changes its output formatting.