NL2REGEX: Distributed Regex Engine

Reading
172 backend tests · ADR 0004 pays JVM startup once per worker, not per task
Verdict
Measured · holds

Distributed Natural Language to Regex Engine for Large-Scale Datasets

Distributed natural language to regex engine that transforms complex tabular dataset patterns using plain English, backed by PySpark and Celery with schema-validated triage.

Role
Creator / Full-Stack Engineer
Timeline
2026
Subject
Distributed Systems

Ravicha2/NL2REGEX

172 cases
backend tests over the API, Spark projection, Parquet normalization, and ReDoS safety
3–5 s
Spark JVM startup per task avoided by the singleton session

  • PySpark 3.5
  • Django 5
  • Celery
  • Redis
  • PostgreSQL
  • React 18
  • Docker Compose
The NL2REGEX workspace: the uploaded file list, its parsed table, and the natural-language prompt box.
The NL2REGEX workspace running from the repository at 9c9fe8a — the uploaded file, its parsed table, and the prompt that drives the job.

Source, as quoted

Ravicha2/NL2REGEX9c9fe8aL16–L26

The ReDoS sandbox is load-bearing, not decorative: both catastrophic-backtracking patterns raise instead of hanging the worker.

    def test_catastrophic_backtracking_regex_times_out(self):
        # (a+)+b on all-'a' string forces exponential backtracking (no 'b' to match)
        pattern = r"(a+)+b"
        with self.assertRaises(RegexSafetyError):
            LLMRegexService.validate_regex_safety(pattern)

    def test_nested_quantifier_backtracking(self):
        # Another pathological pattern: (a|a)* on repeated 'a's
        pattern = r"(a|a)*b"
        with self.assertRaises(RegexSafetyError):
            LLMRegexService.validate_regex_safety(pattern)

tests_regex_safety.py#L16-L26 @ 9c9fe8a

Core intuition & friction

Enable safe, scalable natural-language data transformation without transmitting raw dataset records to LLMs.

The spark and the initial hypothesis

Non-technical analysts and data engineers need to sanitize and reformat massive tabular datasets using natural language without writing fragile regex patterns or uploading sensitive multi-gigabyte files to cloud LLM APIs.

The naive failure mode

Directly prompting an LLM with raw CSV rows causes catastrophic column hallucination, huge token costs, data leakage risks, and catastrophic backtracking (ReDoS) from unverified regex output.

The root problem encountered

Schema hallucination, ReDoS lockups, and memory exhaustion when scaling naive LLM data transformations.

  • LLMs inventing column names not present in uploaded CSV headers (Schema Hallucination)
  • Generated regexes triggering catastrophic backtracking (ReDoS) and freezing Celery worker threads indefinitely
  • Multi-gigabyte file uploads causing out-of-memory (OOM) crashes in pandas workers
  • Zero upload of dataset records to LLM APIs (privacy constraint)
  • Worker stability under concurrent multi-user transformations on large datasets

Why built this way

Key architectural insight

A two-stage pipeline separates schema verification from regex generation, while canonical Parquet normalization enables PySpark distributed execution without worker memory blowups.

Two-Stage LLM Triage

vs Single-prompt end-to-end generation

Stage 1 validates target columns and transformation intent against verified schema headers; Stage 2 generates and validates regex syntax. Eliminates hallucinated columns.

Canonical Parquet Normalization (ADR 0002)

vs In-memory Pandas DataFrames

Uploaded CSVs are immediately converted to partitioned Parquet files, allowing PySpark to execute regex projections in parallel across partitions.

Local Singleton JVM Session (ADR 0004)

vs Per-task Spark session creation

Celery worker processes share a persistent PySpark JVM instance, eliminating multi-second SparkSession initialization latency.

Guardrails and defensive design

  • Regex pre-compilation wrapped in POSIX signal.alarm timeout sandbox to prevent ReDoS
  • SHA-256 prompt hash caching in Redis to prevent redundant LLM invocations
  • Strict Pydantic payload validation on all Celery task inputs

Outcomes & verification

Verification benchmarks

  • 172-test backend suite covering the API surface, Spark projection, Parquet normalization, pagination, and ReDoS-safety cases
  • Validated regex execution safety against known ReDoS malicious payload benchmarks

Real-world impact

  • Lets non-technical users transform large tabular datasets without writing regex or moving records to a cloud LLM.
  • Documented in 5 architectural decision records (ADRs) covering LLM triage, Parquet normalization, and the PySpark session strategy.

Core engineering takeaway

Separating intent extraction from data execution isolates LLM non-determinism from high-throughput distributed engines.

Get in touch

Master of IT at UNSW Sydney (WAM 83 / Distinction, graduating Dec 2026) · Open to full-time Applied AI & Backend Systems roles.

palm.ravicha@outlook.com

EmailCV (PDF)GitHubLinkedInSydney, Australia