Tag: stubborn fanatic

I asked my own law database for section 5 of the Hyderabad Metropolitan Development Authority Act, 2008. It told me there was no such section.

There is. It is on the printed page in front of me. The database had the text, all of it, and could not find it, because the text was filed under a different number.

This is a post about building a searchable corpus of Indian statutes and case law on my own machine, and about the fact that the dangerous failure is never the database that says nothing. It is the one that answers with confidence and is wrong.

I built it over several weeks with Claude. I made the decisions, ran everything on my own hardware, and checked the law myself; the scripts and most of the analysis came from the model. The mistakes are in here as well, because on a project of this shape the mistakes are the content. Nearly every genuine improvement came from something being printed to screen that looked wrong.

Why hold law locally at all

I am an advocate in Hyderabad. My practice is property, title due diligence, drafting. The work needs the text of statutes, rules, government orders and judgments, constantly, and mostly for a narrow set of jurisdictions: Central Acts, Telangana, and the Andhra Pradesh that Telangana inherited.

The obvious answer is a subscription database. They work. But two things pushed me the other way.

The first is cost and dependence. The second matters more: I cannot audit what I cannot see inside. When a subscription database returns nothing for a section, I have no way of knowing whether the section does not exist, or whether their ingestion mangled it the way mine mangled HMDA section 5. When my own corpus does that, I can go and look.

There is also a speed argument that turns out to be real. Searching a million judgments on my own machine takes about a second.

What the corpus is made of

The whole thing lives at D:AiTools on a Windows workstation, a Ryzen 9950X3D with 64 GB of RAM. The pipeline is deliberately dull: scrape original sources, convert to markdown, write parquet, query with DuckDB. No database server, no index server, nothing to keep running. Files on a disk.

An early decision shaped everything after it. There is a bulk dataset of Indian legislation on Hugging Face, and I downloaded it. Then I decided not to use it for statutes, and to scrape India Code and the Telangana Law Department fresh instead. Not because the dataset is bad, but because I wanted to know the provenance of every section I might cite, and to be able to go back to the page it came from. I kept the dataset to cross-check against, once.

What the Act surface holds today, counted this morning:

58,381 rows across eight parquet files

  India Code       828 Central Acts     35,390 sections
                   112 Telangana Acts    1,542 sections
  Telangana Law Dept   184 Acts
  Schedules            380 Acts          1,484 schedule rows
  Act-level PDFs       452 Acts
      196  Andhra Pradesh, pre-2014, applies in Telangana
      179  Telangana
       56  Andhra Pradesh, post-2014, residuary AP only
       18  Central
  Harvested G.O.s, circulars, orders   757 rows

Separately there is an index of my own prior work: 18,932 documents out of my OneDrive, read-only, so a search can turn up the deed I drafted in 2019 alongside the section it relied on.

That AP split is not decorative. Andhra Pradesh was bifurcated on 2 June 2014, and an AP Act from before that date still applies in Telangana while one from after it does not. Every result line says which, because an Act that does not apply is still an Act, and the distinction has to survive into whatever is filed.

Counting is not checking

Here is the mistake that reshaped how I check this corpus.

The corpus held the HMDA Act with 58 sections, numbered up to 58. That looks complete. It is not. Sections 5, 6 and 8 were absent, and their text was inside the row filed as section 9, a row that opens with the repeal of a 1988 water-rates ordinance and runs to 10,362 characters, having swallowed several sections that come before it.

The cause is visible on the printed page. HMDA prints its section headings as marginal notes, in a column beside the body text. Extract a text layer from that PDF and the headings interleave with the sentences, so 5. never begins a line, and a splitter looking for a number at the start of a line never sees one.

So I stopped trusting counts and audited numbering continuity instead: for every Act holding eight or more numbered sections, is the run of numbers unbroken?

When that ran, on 27 August 2026, it found 1,151 Acts with eight or more sections and 99 with a gap. Most gaps are honest: sections repealed, or omitted from the source. But in 38 of them the missing section’s heading was present inside another row. The text was in the corpus, filed under the wrong number.

Re-running the same audit today gives 1,158 Acts and 86 gaps. The difference is the repair described below.

The consequence is worth stating plainly. Full-text search reaches misfiled text without difficulty. It is a lookup by section number that fails, and it fails by reporting that a section does not exist when it does. That is the wrong direction to fail in.

repair.py, and five ways it nearly filed the wrong text

Recovering misfiled sections sounds simple: find the missing number used as a heading inside another row, cut from there to the next heading, file it under its own number.

It is not simple, and the first version was wrong in five separate ways. All five were found by running it in a sandbox and reading the output before it touched anything.

It missed the section it was written for. The heading pattern required text to continue on the same line. India Code prints 5. alone, with (1) There shall be... on the line below. HMDA section 5, the whole reason the script existed, did not match.

Cuts ran to the end of the host row. A recovered section 8 swallowed section 9 and everything after it. Cuts now end at the next heading of any number.

Footnotes were recovered as sections. 54. Section 43 substituted by Act No.11 of 1969 is a footnote. It looks exactly like a section heading.

Page-group rows were used as hosts. Where an Act could not be split reliably it is stored as page groups, which contain the entire text, so every number appears in them. Recovering from those produces nonsense.

The same number was recovered from six different rows of the Wealth-tax Act.

The gate that did most of the work is a structural one: a row can only have swallowed sections that precede it. So a recovered number must be lower than its host’s number and within 40 of it. Without that gate, the Greater Hyderabad Municipal Corporation Act “recovered” section 19 out of a row filed as section 687, which is a schedule of cycle-rickshaw fees.

Ungated, the pass proposed 34 Acts and 133 sections. Gated, it proposed 23 Acts and 49 sections, and the 23 rejected candidates each carry a stated reason.

Today repairs_acts.parquet holds 49 recovered sections across 23 Acts, each labelled with the row it was cut from and marked medium confidence. It never edits another script’s file. The recovered rows join the same glob the reader already reads.

A correction

An earlier note of mine recorded that the GHMC Act had “37 of 53 gaps recoverable”. That was wrong, and it came from a crude heading search with none of the gates above.

A properly gated pass recovers nothing from GHMC. Its numbering gaps are footnote markers and schedule item numbers, not missing sections. Today the audit still shows GHMC with 635 sections present and 52 numbers missing, and that is the correct answer: they were never sections.

I am recording the wrong figure as well as the right one because a corpus you cannot correct in public is a corpus nobody should trust, including you.

Heuristics that fail interestingly

Three gates keep schedules and tables out of the recovery pass, and each was calibrated against real documents.

Digit density. Statutory prose runs 3 to 4 per cent digits. Schedules run 24 per cent and up. The threshold sits at 18. This caught a recovered “section” of 247,454 characters that was the Telangana Panchayat Raj Act’s schedule of village names.

Line length. Text laid out as a table puts each field on its own line. Village schedules average 9 characters per line; statutory prose averages 75.

That second gate was first set at 28 characters, and it rejected HMDA sections 5 and 8. India Code wraps its prose at about 27, and a section that is a list of committee members has short lines. A threshold tuned on the wrong sample rejected precisely the sections the whole exercise existed to recover. Moved to 15, the pass went from 44 sections to 49, the five additions being HMDA sections 5 and 8, Telangana Municipalities section 32, and two sections of the Telangana State Private Universities Act.

Capture ratio. Any split that captures under 92 per cent of a document’s text is rejected in favour of page chunks. This exists because of a specific failure: an Act splitter applied to a Government Order treats the numbered list of earlier G.O.s it cites as the sections. G.O.Ms.No.59 came out as a single “section 4” holding 47 per cent of the document, with the abstract and the operative order silently gone. Instruments that are not Acts, Rules or Regulations are no longer split at all.

Fifty gigabytes of judgments, and what they do not carry

Judgments come from the open-india-law dataset: 63 parquet files, 53.7 GB. I hold all of them.

The indexed corpus covers the courts that matter to my practice:

  telangana        1,005,147 judgments
  andhra-pradesh     261,778
  supreme-court       37,014

Now the part that decides how you use this data. Metadata completeness, measured this morning:

  court            judgments    no citation   no judges
  telangana        1,005,147        100.0%       81.9%
  andhra-pradesh     261,778        100.0%       88.3%
  supreme-court       37,014          0.0%        0.0%

Every Supreme Court judgment carries a citation and a bench. Not one High Court judgment in over 1.26 million carries a citation. You get the text and the case number and nothing you can put in a plaint.

That is not a defect in the dataset so much as a fact about what is freely available, and it changes the workflow rather than ending it. The corpus is where I find the case and read the reasoning. The citation comes from elsewhere, and it comes before anything is filed.

The dates need care too. Telangana decision dates in this data run from 23 October 1952 to 27 December 2079. Twelve judgments are dated in the future. The year column in the Andhra Pradesh file tracks ingestion rather than decision and cannot be used for anything.

Which court binds

For a matter in Telangana, a Supreme Court judgment binds, the Telangana High Court binds, and the Andhra Pradesh High Court binds only for the period before bifurcation. Bombay and Madras are persuasive, and worth reading, and they are not the same thing. The distinction has to survive into the document.

The dataset splits along that line almost exactly, which was a pleasant surprise:

  telangana, decided before 01.01.2019          707,308
  andhra-pradesh, decided before 01.01.2019       1,011
  andhra-pradesh, decided 01.01.2019 onwards    259,200

The in_telangana file is the predecessor court’s whole corpus going back to 1952. The in_andhra-pradesh file is overwhelmingly the reconstituted court at Amaravati. So the default tier applies a date predicate to the AP file alone, and the binding tier comes to 1,043,172 judgments. Every result line is labelled BINDING or persuasive, and a binding search says on its face that the other High Courts were not searched.

This nearly went wrong in a way worth describing, because it is the most dangerous class of bug in the whole project.

DuckDB’s regexp_extract(string, pattern) returns the whole match, not the capture group. The build stored in_telangana_judgments where it meant telangana. Every comparison against a bare court name then failed silently. The effect was that 260,766 post-2019 Amaravati judgments sat in the binding tier, labelled BINDING, and the authority ranking collapsed to plain date order.

Nothing errored. Nothing looked broken. A one-argument mistake quietly relabelled persuasive authority as binding, which in a document going to court is exactly the sort of error you do not get to explain away. The build now passes the group index, and the reader normalises whatever form it finds, so a stale index cannot mislabel authority even if it is rebuilt wrongly.

Search without a search engine

The obvious way to search a million judgments is a full-text index. I could not use one, for a reason that is specific and worth explaining, because it shaped the design.

The environment Claude reaches my machine through is a small sandbox: 2 CPUs, 3 GB of RAM. My workstation reports 32 CPUs. A full scan of the 9.1 GB binding tier takes seconds on the workstation and over three minutes in the sandbox, which is past the timeout on a single call. DuckDB’s full-text extension cannot be installed in that sandbox at all: the proxy variable is malformed for its parser, and with the proxy unset there is no network.

So an index that needed the extension would have been unreadable by the one reader that most needed it. The answer was plain sorted parquet, built natively on the fast machine, read cheaply by anything.

Four files:

  cases.parquet      one row per judgment            0.05 GB
  text.parquet       chunks, sorted by case          4.24 GB
  vocab.parquet      words and their ids             0.10 GB
  postings.parquet   (token, case, chunk)            0.91 GB
                                            total    5.31 GB

Over 1,303,939 judgments, 15,057,895 distinct indexed words, and 397,445,969 postings.

Two calibration decisions matter more than the structure.

Minimum word length is 4, not 5. Sale, deed, gift, rent and lien all carry legal phrases. Three-letter words are lost, so lis pendens is found on pendens alone, which is fine because the rarer half of a phrase does the work.

The commonness cut is 10 per cent, not 1. Words appearing in more than that share of cases are not indexed, because indexing them costs enormously and narrows nothing. The first threshold chosen was 1 per cent. Measured against 17,684 Supreme Court cases:

  possession      5.3%
  registration    1.9%
  performance     1.8%
  adverse         1.6%
  injunction      1.1%
  mortgage        1.0%

A 1 per cent cut would have excluded every one of them. It would have produced a technically excellent index, of a corpus of property judgments, that could not search for possession, registration, performance, adverse or mortgage. Only 375 words are common enough to exclude at 10 per cent, and when a search hits one it is reported by name, so a search for “part performance” says it searched on performance alone.

Matching is by proximity, not merely co-occurrence. Postings carry the chunk as well as the case, so intersecting on chunk requires the words to fall within the same passage of roughly 2,000 characters. It does not prove the phrase appears in that order. It is much better than “both words are somewhere in this judgment”, and the phrase still has to be confirmed by reading.

Three performance facts

Row group size is the difference between a skip and a scan. At 50,000 rows per group, each group in text.parquet held about 80 MB of text, so fetching 200 scattered cases decompressed most of the file and took 29 seconds. At 1,000 rows a group is under a megabyte and the reader skips almost everything.

An IN-list of literals is pushed down; a JOIN is not. The same 200 lookups written as a join against a temporary table read the entire file: 0.1 seconds became over 40.

A warm cache lied to me twice. Verifying a phrase by reading the text costs about 20 seconds cold, however few cases it asks for, because a 4 GB store cannot stay resident in 3 GB of RAM. Two earlier measurements of 2.5 seconds were a warm cache, and they misled the design twice before the proximity approach removed the need to read text at query time at all.

Where it landed, on 2 threads:

  specific performance   1.02s   8,466 cases
  adverse possession     0.91s
  lis pendens            0.85s
  pattadar pass book     0.92s

Against over three minutes before any of this.

One further thing had to be designed rather than discovered. Ordering strictly by authority, a window on “specific performance” returned 400 Supreme Court cases and nothing at all from 11,429 Telangana candidates. Telangana is the court I appear before. Display slots are now shared between courts, allocated smallest tier first so slack flows onward.

What I still do not trust

A corpus is only as useful as its own account of where it is weak.

Two Central Acts cannot be completed from India Code. The Income-tax Act 1961 and the EPF Act 1952. India Code holds no Act-level PDF for them and only a fraction of the sections. Both are fetched separately.

The Hindu Succession (Andhra Pradesh Amendment) Act 1986 matters for coparcenary devolution before 9 September 2005, and the only copy I have is from PRS Legislative Research, which prints its own disclaimer that the text is unverified. It is deliberately kept off the Act surface and findable only in my own-work index. The real text has to come from the Law Department before it is cited.

Every HMDA document arrived over an unvalidated TLS connection. All 75 are stamped tls=unverified, because hmda.gov.in’s certificate chain does not validate. They are usable and they are flagged, and any one of them is checked against the browser copy before it is cited.

OCR misreads digits. 522 scanned PDFs, 6,738 pages, English and Telugu, banded 222 high confidence, 193 medium, 107 low. A document number came back as NO,P 2418/2013-14. Any survey number, extent or consideration taken from OCR text is checked against the original PDF. Results drawn from OCR print a warning saying so. And the opening pages of a registered deed are the stamp paper, which OCRs as ornament noise, so an OCR’d deed is judged by its body and not its first lines.

Some sources cannot be harvested at all. The CCLA site renders a page header and stops: viewActs.do, viewGO.do and viewCirculars.do each return two links and no table. That is not a selector problem, the pages are empty. And goir.telangana.gov.in is an ASP.NET WebForms application with view-state and cascading postbacks, where a result set only exists after a POST carrying the view-state the server has just issued. A listing-page harvester cannot touch it.

A naming rule learned the hard way. A document’s title becomes its act name, and lookups match that name loosely. A document titled “Registration Rules, Telangana — rules under the Registration Act, 1908” shadowed the Registration Act itself, so asking for section 49 of the Registration Act returned rule 49 instead of section 49, which is the provision on the effect of non-registration. A title must never contain another instrument’s name.

And one Act answers twice, by design. HMDA section 5 exists in the corpus in two forms: the Act as originally enacted, naming APIIC, APHB and APSRTC, hand-read by me from printed pages; and the India Code text as adapted for Telangana by G.O.Ms.No.45, Law (F) Department, dated 1 June 2016, naming TSIIC, the Telangana Housing Board and TSTRANSCO. Both are correct. For the composition of the Executive Committee as it stands today, the adapted text governs.

What this is actually for

It is not a product and I am not proposing that anyone else run one. It is a tool with a known shape, whose weaknesses are written down where I can see them, which answers in about a second, and which does not stop working if a subscription lapses or a website is redesigned.

The thing I did not expect was how much of the work would be auditing rather than building. The scraping was a weekend. The gates, the calibration, the gap analysis, the five defects in the recovery pass and the one-argument bug that mislabelled a quarter of a million judgments as binding — that was the actual project.

Which brings it back to HMDA section 5. It was in the corpus the whole time. The lesson is not that the pipeline was bad. It is that a lookup which answers “no such section” and a lookup which answers “here is section 5” can both be produced by a database that holds the same bytes, and the only thing standing between them is whether anyone checked.

Glossary

This post has two audiences, and each will find half of it obvious. The terms below are defined in the order they appear, on the assumption that no reader knows both halves.

Act surface — my term for the set of files a section lookup searches, as distinct from the full-text library. A document promoted to the Act surface can be asked for by section number. One left in the library can only be found by searching its words.

Parquet — a file format for storing tables in columns rather than rows, compressed, with statistics recorded for blocks of rows. A reader can consult those statistics and skip blocks that cannot contain what it wants, without decompressing them.

Row group — a block of rows inside a parquet file, and the unit that a reader skips or reads. Their size is a design decision, discussed above.

DuckDB — a database engine that runs inside your program rather than as a server, and reads parquet files directly from disk. No installation to maintain, no daemon running.

Predicate pushdown — the behaviour that makes the above work: the filter is handed down to the file reader so it can skip data, instead of the reader returning everything and the filter discarding most of it afterwards. An IN-list of literal values is pushed down. A join against another table generally is not.

Full-text index / FTS — the conventional machinery for searching text at speed, usually a database extension. Not used here, for the sandbox reasons set out above.

Vocabulary and postings — the two halves of any word index. The vocabulary maps each word to a number. The postings list records, for each word number, where it occurs. Searching is intersecting postings lists.

Posting — one entry in that list. Here each posting is a triple of word, case and chunk, which is what makes proximity matching possible.

Chunk — a passage of a judgment, roughly 2,000 characters. Judgments are stored in chunks so that “these two words appear in the same chunk” can stand in for “these two words appear near each other”.

Proximity matching — requiring search terms to fall within the same chunk, rather than merely somewhere in the same document. It does not prove the words appear as a phrase in that order.

MINLEN — the shortest word the index will store. Set to 4 here.

Commonness cut — the share of documents above which a word is too common to be worth indexing. Set to 10 per cent here, for the reasons measured above.

OCR — optical character recognition, deriving text from a scanned image. Reliable for prose, unreliable for digits, which is why numbers taken from OCR are checked against the original.

Text layer — the machine-readable text embedded in a PDF. A PDF produced from a word processor has a good one. A PDF that is a photograph of a page has none, and needs OCR.

Marginal note — a section heading printed in the margin beside the body text rather than above it. The cause of the HMDA failure: in the text layer the marginal notes interleave with the sentences.

Splitter — the code that divides a statute into its sections. Most of the failures described here are splitter failures.

Page group — the fallback when a document cannot be split reliably: it is stored as groups of pages, flagged as unsplit, rather than given section numbers that might be wrong.

Capture ratio — the proportion of a document’s text that a split accounts for. Below 92 per cent the split is rejected, so no document can silently lose text.

Digit density — the proportion of characters in a block that are digits. Prose runs 3 to 4 per cent, schedules 24 per cent and up.

Harvester — the script that fetches documents from government websites, driven by a manifest describing each source, rather than a separate scraper per site.

Manifest — the configuration file listing each source, its pages, and how to treat what it returns.

G.O. — Government Order. A departmental instrument. G.O.Ms. is an Ordinary Order; the number and date identify it.

India Code — the Government of India’s official repository of Central and State legislation.

Bifurcation — the division of Andhra Pradesh on 2 June 2014, creating Telangana. An Andhra Pradesh Act from before that date continues to apply in Telangana; one from after it does not.

Binding and persuasive — a judgment of a court above you in your own hierarchy binds. A judgment of a court of equal standing elsewhere is persuasive: worth reading, worth citing, but it does not compel a result.

Pattadar pass book — the Telangana land record document evidencing a cultivator’s rights. Used above as a search example because it is a phrase of exactly the shape this index handles well.

Coparcenary — in Hindu law, the body of family members who take an interest in ancestral property by birth. Relevant here because devolution before 9 September 2005 turns on statutory amendments that must be sourced accurately.

Sandbox — the constrained environment the AI assistant reaches my machine through: 2 CPUs, 3 GB of RAM, restricted network. Several design decisions exist only because of it.

TLS validation — the check that a website’s certificate chains to a trusted authority. tls=unverified on a stored document means the download was made without that check succeeding, and the document should be confirmed against the browser copy before it is relied upon.