# NL2REGEX: Distributed Regex Engine > Distributed Natural Language to Regex Engine for Large-Scale Datasets **Status:** Measured to an artifact. **Role:** Creator / Full-Stack Engineer. **Timeline:** 2026. **Category:** Distributed Systems. Distributed natural language to regex engine that transforms complex tabular dataset patterns using plain English, backed by PySpark and Celery with schema-validated triage. ## Measurements | Value | What it measures | | --- | --- | | 172 cases | backend tests over the API, Spark projection, Parquet normalization, and ReDoS safety | | 3–5 s | Spark JVM startup per task avoided by the singleton session | ## The artifact that settles this The ReDoS sandbox is load-bearing, not decorative: both catastrophic-backtracking patterns raise instead of hanging the worker. Quoted from `Ravicha2/NL2REGEX` at `tests_regex_safety.py#L16-L26` (9c9fe8a), `backend/tests/jobs/tests_regex_safety.py#L16-L26`, verbatim: > def test_catastrophic_backtracking_regex_times_out(self): > # (a+)+b on all-'a' string forces exponential backtracking (no 'b' to match) > pattern = r"(a+)+b" > with self.assertRaises(RegexSafetyError): > LLMRegexService.validate_regex_safety(pattern) > > def test_nested_quantifier_backtracking(self): > # Another pathological pattern: (a|a)* on repeated 'a's > pattern = r"(a|a)*b" > with self.assertRaises(RegexSafetyError): > LLMRegexService.validate_regex_safety(pattern) Permalink: ## The intuition Enable safe, scalable natural-language data transformation without transmitting raw dataset records to LLMs. - The spark: Non-technical analysts and data engineers need to sanitize and reformat massive tabular datasets using natural language without writing fragile regex patterns or uploading sensitive multi-gigabyte files to cloud LLM APIs. - The naive failure mode: Directly prompting an LLM with raw CSV rows causes catastrophic column hallucination, huge token costs, data leakage risks, and catastrophic backtracking (ReDoS) from unverified regex output. ## The problem Schema hallucination, ReDoS lockups, and memory exhaustion when scaling naive LLM data transformations. **Constraints** - Zero upload of dataset records to LLM APIs (privacy constraint) - Worker stability under concurrent multi-user transformations on large datasets **Edge cases handled** - LLMs inventing column names not present in uploaded CSV headers (Schema Hallucination) - Generated regexes triggering catastrophic backtracking (ReDoS) and freezing Celery worker threads indefinitely - Multi-gigabyte file uploads causing out-of-memory (OOM) crashes in pandas workers ## Why it was built this way Two-stage LLM schema validation, Parquet normalization, PySpark distributed execution, and ReDoS timeout sandboxing. A two-stage pipeline separates schema verification from regex generation, while canonical Parquet normalization enables PySpark distributed execution without worker memory blowups. **Trade-offs** - Two-Stage LLM Triage — Stage 1 validates target columns and transformation intent against verified schema headers; Stage 2 generates and validates regex syntax. Eliminates hallucinated columns. (against: Single-prompt end-to-end generation) - Canonical Parquet Normalization (ADR 0002) — Uploaded CSVs are immediately converted to partitioned Parquet files, allowing PySpark to execute regex projections in parallel across partitions. (against: In-memory Pandas DataFrames) - Local Singleton JVM Session (ADR 0004) — Celery worker processes share a persistent PySpark JVM instance, eliminating multi-second SparkSession initialization latency. (against: Per-task Spark session creation) **Guardrails** - Regex pre-compilation wrapped in POSIX signal.alarm timeout sandbox to prevent ReDoS - SHA-256 prompt hash caching in Redis to prevent redundant LLM invocations - Strict Pydantic payload validation on all Celery task inputs ## Outcomes Fault-tolerant distributed regex processing documented in 5 ADRs and a 172-test backend suite. **How it was verified** - 172-test backend suite covering the API surface, Spark projection, Parquet normalization, pagination, and ReDoS-safety cases - Validated regex execution safety against known ReDoS malicious payload benchmarks **Impact** - Lets non-technical users transform large tabular datasets without writing regex or moving records to a cloud LLM. - Documented in 5 architectural decision records (ADRs) covering LLM triage, Parquet normalization, and the PySpark session strategy. **Takeaway:** Separating intent extraction from data execution isolates LLM non-determinism from high-throughput distributed engines. ## Stack PySpark 3.5, Django 5, Celery, Redis, PostgreSQL, React 18, Docker Compose ## References - github: --- ## Fetching the rest of this site - This page as HTML: - Concise index: - Full dossier: - When to use this material: - API description (OpenAPI 3.1): - API reference: - Sitemap: