Prepare documents for AI

Step 1

Inventory and scope

Why You can only prepare a collection you have listed. Indexing everything also pulls in drafts, personal mail, and files a smaller group was meant to see alone.

How

• List every place documents live: shared drives, SharePoint or Google Drive exports, wikis, ticket exports, local project folders, and mail archives.
• For each collection, write the path, the kinds of files, the date range, the owner, and whether people still treat that place as the source of record.
• Mark each collection in or out. Out includes drafts you will not stand behind, personal mailboxes, copies of a system you already included, and anything legal or contract staff say must stay out.
• Name one owner for the prepared corpus, and keep the owner of each source collection. Someone has to answer “is this the current file?”

A spreadsheet is enough. Suggested columns: location, file types, date range, owner, in or out, reason, sensitivity. This is a decision record. It does not need to be a cataloging system.

Step 2

Clean up

Why Duplicates and old drafts compete with the current document. Search then returns several answers, and the longest one is often the wrong version.

How

• Find exact copies by file content. fdupes groups files that are byte-for-byte identical. Keep one copy of each group.
• Find near copies by hand where the names give them away: final, v7, latest, dates, and (1). Ask the owner which file is current. If nobody will say, keep the latest approved copy and write that rule in the inventory.
• Move rejected copies to a folder you do not index, or delete them when retention rules allow. Leave them out of the folder you will convert.
• Remove junk an indexer would treat as documents: .DS_Store, Thumbs.db, desktop.ini, Word lock files (~$*), .tmp files, empty files, and installer logs.

fdupes -r ./documents

Step 3

Folders and file names

Why The path becomes part of the citation. A person and a retrieval system both need to see which document a passage came from.

How

• Use one collection per top folder: policies, contracts, meetings, support.
• Stay shallow. Collection, then year or topic, then the file is enough.
• Use lowercase names with hyphens. Put an ISO date first when the date identifies the document: 2024-11-01_travel-policy.md.
• Leave final, latest, and new out of names. The current folder holds the current files. Older copies live in an archive folder you do not index.
• Add a short README in each collection: what it contains, who owns it, and what “current” means.
• Keep secrets and personal data out of file names. Names show up in logs, citations, and search results.

policies/travel/2024-11-01_travel-policy.md
meetings/2024/2024-03-12_board-minutes.md
contracts/acme/2023-06-01_master-services-agreement.pdf
mail/2024/thread-9f3a/

Step 4

Metadata

Why Title, date, author, source, type, and permissions tell you what a retrieved paragraph is and who may see it. A file’s modified time is a weak substitute for the date on the document.

How

Put a header at the top of each text file, or a sidecar with the same stem, such as travel-policy.meta.txt. Use the date printed on the document when it differs from the file timestamp.

• title — the name a person would use.
• date — the document date, as YYYY-MM-DD.
• author — a person or the team that issued it.
• source — where this copy came from: a URL, a library path, or a message id.
• type — a short list you actually use: policy, contract, email, minutes, spreadsheet, how-to.
• permissions — the audience: public, internal, or a named group such as hr.

Keep a stable id if you rename the file, so a later update replaces the same document. A permission label does nothing until the search application filters on it. Step 11 is where that filter has to hold.

---
title: Travel policy
date: 2024-11-01
author: People team
source: sharepoint://policies/travel.docx
type: policy
permissions: internal
id: travel-policy
---

Step 5

PDFs

Why A PDF is a picture of a page plus, sometimes, a hidden text layer. If the text layer is missing or scrambled, extraction returns blank pages or nonsense, and the model has nothing true to quote.

How

Text PDFs and scanned PDFs

Open the file and try to select a sentence. If you can copy real words, it is a text PDF (sometimes called born-digital). If selection fails, or the copied text is gibberish, the page is a scan or a broken export.

Confirm from the command line with Poppler’s pdftotext. A text PDF prints words. A scan prints almost nothing.

pdftotext -eol unix policy.pdf policy.txt

Default pdftotext rebuilds reading order, which is what you want for prose. Add -layout when you need to see columns while you check a page. -layout pads lines with spaces, so it is a poor file to chunk as-is. Page breaks are form-feed characters; leave them in if you want to cite a page number. -nopgbrk removes them.

Some PDFs have a text layer that is still wrong, because the font encoding is damaged. If the words look garbled, treat the file like a scan and OCR it.

OCR with Tesseract and OCRmyPDF

Tesseract reads an image of text. OCRmyPDF runs Tesseract on a PDF and writes a searchable PDF with the text hidden behind the page image. Keep the original scan, and write the OCR result beside it.

Set the language. Quality drops when the language is wrong. Combine languages with a plus sign when a document mixes them. --deskew straightens pages that were scanned slightly crooked. --sidecar also writes a plain-text file.

ocrmypdf -l eng --deskew --sidecar policy.txt scan.pdf policy-searchable.pdf

OCRmyPDF stops when a PDF already has text, so a mixed file (some real text, some scans) needs --skip-text. That option OCRs only the pages that have no text yet.

ocrmypdf -l eng --skip-text mixed.pdf mixed-searchable.pdf

For one image, call Tesseract directly. For a PDF of many scans, prefer OCRmyPDF so the text stays tied to pages.

tesseract page.png stdout -l eng

Install the Tesseract language packs you name with -l. After OCR, read a few pages. Skew, handwriting, stamps, and tiny footnotes still fail, and those pages need a person.

Tables and layout

Simple extractors read left to right. Two columns become one scrambled column. A table becomes a sequence of cell values with the row and column breaks gone. Headers and footers get glued onto the first and last line of every page.

For a report whose meaning is in the tables, use a tool that emits a Markdown table or structured cells, then spot-check every important table against the PDF. If a table is the contract, keep the PDF page number in the text so someone can open the original.

Extraction tools

• pdftotext (Poppler). Fast, and the right start for ordinary text PDFs. Weak on complex layout.
• PyMuPDF. A Python library for page-by-page text, images, and positions. Use it when you want page numbers and a script you control.
• Docling. Converts PDFs and several office formats to Markdown or JSON, with attention to reading order and tables. Current docs show docling convert policy.pdf --to md; older releases used docling policy.pdf. Trust docling --help on your install.
• Unstructured. Splits a document into elements such as title, paragraph, and table, which are natural boundaries for chunking.
• Marker. Converts a PDF to Markdown in one pass, including many tables and multi-column pages.

Start with pdftotext for simple text PDFs. Move to Docling, Unstructured, or Marker when columns or tables come out wrong. A short PyMuPDF script is a good middle step when you only need page-labeled text:

import pymupdf

doc = pymupdf.open("policy.pdf")
for number, page in enumerate(doc, start=1):
    print(f"\n\n--- page {number} ---\n")
    print(page.get_text())

Step 6

Emails

Why Decisions often live in mail, and a raw export is full of signatures, legal footers, and the same quoted history repeated under every reply. Indexed that way, one thread crowds out everything else.

How

Export .eml, .mbox, or .pst

• .eml is one message per file. It is the easiest form to process and to cite.
• .mbox is many messages in one file. Thunderbird and Google Takeout mail use it.
• .pst is an Outlook archive. Convert it before ingestion. readpst (from libpst) can write one file per message with an extension such as .eml.

readpst -e -o mail/ archive.pst

Keep the exported originals. Build a separate text copy for the index.

Keep threads together

Group messages with the Message-ID, In-Reply-To, and References headers. Sort each thread by date, and write one Markdown file with a heading per message. When those headers are missing, group by subject after you strip prefixes such as Re:, Fw:, Fwd:, and Aw:, and expect a few unrelated messages to land together. Check those.

# Q1 pricing
permissions: internal

## 2024-02-03 Ada Lovelace to Ben Smith
message-id: <abc@example.com>

Can we hold last year's rate through March?

## 2024-02-04 Ben Smith to Ada Lovelace
message-id: <def@example.com>

Yes, through 31 March. The attached schedule is the one to use.

Attachments

Save attachments next to the parent message, with the message id or date in the file name. List those file names in the message text so the link survives. Convert or OCR an attachment the same way you would if it had arrived on its own: a contract PDF follows step 5, a sheet follows step 8. Skip logos, tracking pixels, and decorative images, and note that you skipped them.

Signatures, disclaimers, and quoted replies

Clean the text copy, and leave the .eml untouched.

• Drop lines that start with >. They repeat an earlier message that you already store once in the thread file.
• Drop a trailing block that begins with the conventional signature mark -- followed by a space, when that mark is present.
• Drop legal footers you have listed, once you can see the same paragraph on message after message.

Read a sample of threads after the cleaner runs. Some clients quote without >. Sometimes the only statement of a decision sits in a signature or a quoted blob. Fix those by hand, or rely on the thread file, which already contains the earlier message a single time.

Keep sender, recipient, and date

Copy From, To, Cc, Date, Subject, and Message-ID into the text header. The body cleaner has to leave those fields alone. Bcc is often missing on mail you received; leave it out rather than guessing. These fields are how someone checks an answer against the real message.

Step 7

Text, Word, and office files

Why A .docx or .odt file is a package of XML. Models need the words and the headings. The wrong encoding turns accents into junk and can split a word so search misses it.

How

Convert Word, ODT, RTF, and HTML to Markdown so headings stay headings. Pandoc is the general tool. --wrap=none avoids hard line breaks in the middle of sentences.

pandoc -f docx -t markdown --wrap=none -o policy.md policy.docx

MarkItDown converts Office files to Markdown for text pipelines, including Word, PowerPoint, and Excel. Use it when you want one command across those formats. For slides, see step 9. For a large sheet, prefer the CSV rules in step 8.

markitdown policy.docx -o policy.md

Use plain text when the source is already prose with no headings. Keep the original Office file if people still edit there. The Markdown is the build you index.

Encoding and line endings

• Save every text file as UTF-8.
• On Linux and many other Unix systems, file --mime-encoding prints a guess. Open the file and look at accents, quotes, and dashes before you trust the guess.
• Convert a known legacy encoding with iconv. Let the command fail if the bytes do not match, so a bad guess is visible.
• Use LF line endings (\n). dos2unix converts a file from CRLF. In Git, a .gitattributes line of * text=auto normalizes text files on commit.

file --mime-encoding notes.txt
iconv -f WINDOWS-1252 -t UTF-8 notes.txt > notes.utf8.txt

Step 8

Spreadsheets and CSV

Why A worksheet is often several tables, a chart, and a title stuffed into one grid. Retrieved row by row, that mix has no stable meaning.

How

• Put one table in each sheet you export. If a sheet stacks two tables under a chart, split them into two CSV files and describe the chart in a sentence, or leave the chart out.
• Use a single header row. Names are short, unique, and readable: invoice_date, not Column1 and not two columns both called Date.
• Unmerge cells inside the table. Keep totals out of the data file. If you need a total, write it in the column note.
• Export each table to CSV in UTF-8. Write dates as YYYY-MM-DD. Store numbers without currency signs, and put the unit in the column name or the note (amount_usd).
• Describe the columns in a short note beside the CSV: what one row represents, what each column means, and the allowed values when they are a small set.
• When the formulas themselves matter, keep the workbook and also export the values.

# invoices.csv
One row is one invoice.

invoice_id   Identifier in the billing system
issued_on    Invoice date, YYYY-MM-DD
amount_usd   Total in US dollars, no currency symbol
status       open, paid, or void

Step 9

Images, slides, and scans

Why A chart, a photo, and a slide made of pictures are invisible to text search until someone describes them or OCR reads the words on them.

How

• Write alt text or a caption for every image that carries meaning. Say what it shows and the figure that matters (“network diagram: failover path is backup-east, added March 2024”). A file name is not a caption.
• Store the caption in a sidecar text file (network-diagram.txt) or under the figure in the Markdown you already produced.
• For slides, write one Markdown section per slide. Include the slide number, the title, the body text, and the speaker notes. MarkItDown reads PPTX files. Add your own caption when a slide is mostly a picture.
• OCR page scans with Tesseract or OCRmyPDF, as in step 5. A photograph of a whiteboard often comes back garbled. Transcribe the parts you need when the OCR text is wrong.

markitdown slides.pptx -o slides.md

Step 10

Chunking

Why Retrieval works on passages, and models read a limited amount of text at once. A whole manual as one blob is hard to retrieve. A single sentence with no heading is hard to understand.

How

• Split on headings and sections first. If a section is still very long, split it into passages of a few hundred words. Test that size with real questions (step 12) before you tune it.
• When a split would cut a sentence or a rule in half, repeat the last sentence or two at the start of the next passage. A large overlap mostly stores the same text twice.
• Put the heading path on every chunk (“Travel policy > Hotels > International”), plus the document title, the source path or URL, and a page or slide number when you have one.
• Keep a small table in one chunk. For a large table, chunk by row and repeat the column header and the table name on each piece.
• Keep a warning with the rule it qualifies. Keep the sentence that introduces a list with that list.
• Give each chunk a stable id, such as travel-policy#hotels-international, so an update replaces that passage instead of adding another copy.

id: travel-policy#hotels-international
source: policies/travel/2024-11-01_travel-policy.md
title: Travel policy
section: Hotels > International
page: 4
permissions: internal

International hotel stays need approval from the traveler's manager
before booking. The cap is the city rate in the finance sheet.

Step 11

Privacy and security

Why A retrieval system quotes what you indexed. Anything in the corpus can show up in an answer, and sending that text to a hosted model is a disclosure. This section is practical hygiene, not legal advice.

How

Decide the audience before you add files. An assistant for one team and a bot on a public site are different corpora. Match the corpus to the people who can query it.

Redact personal data you do not need

Remove national ID numbers, payment card numbers, bank details, home addresses, health details, passwords, and API keys when the document still answers questions without them. Presidio can flag likely personal data. Treat its hits as a review queue. It misses things, and it also flags names you may need to keep, such as the author of a policy. A person reviews high-risk collections.

Honor access rights

If a folder is limited to HR, leave it out of a company-wide index. Store the permission label from step 4, and enforce it in the application that runs the query. A label in the file is not a control on its own. Filters belong in the search layer, where a user only retrieves chunks they are allowed to read.

Leave these out

• Passwords, keys, tokens, and connection strings.
• A full personal mailbox, when the task only needed a shared team inbox.
• Documents whose readers are a smaller group than the people who can query the index.
• Material a contract, a statute, or a legal hold says you cannot process this way. Ask the people who own that decision.
• Customer content your agreement does not allow you to send to a model provider.

Know where the index and the model run. Text sent to a third-party API leaves your network. Prefer a setup that matches the rule you already follow for that data.

Step 12

Quality checks

Why Extraction fails quietly. A script can finish with empty pages, reversed columns, or a running header glued into every chunk.

How

• Spot-check a sample of each file type, including the awkward ones: a scan, a two-column PDF, a spreadsheet, a long email thread, and a slide deck.
• Open the original beside the text. Read the start, a page from the middle, one table, and one page that has a header, a footer, or footnotes.
• Look for empty output, damaged characters (a UTF-8 file read as Latin-1, or the reverse), missing headings, a table read across the row instead of down it, and the same disclaimer repeated on every page.
• Write questions people actually ask. For each one, record the document and the passage that should answer it. Run the questions through search or your retrieval tool.
• On a miss, fix the file or the chunk first. Typical causes are missing text, a bad split, or an old copy still in the index beside the new one. Change the prompt after those are fixed. Keep the question list and run it again after each pipeline change.

Step 13

Keeping it fresh

Why A stale corpus answers with last year’s policy. Two editable copies drift apart the week someone updates only one of them.

How

• Keep a single source of truth. People edit the policy site, the contract folder, or the sheet. The text you index is a build from that source, not a second place to edit.
• When a source changes, re-extract that document and replace its chunks under the same id. Leave the old text out of the index.
• Log each change: date, document id, old and new hash, and who or what job ran it. Git history is enough when the text files live in Git. Otherwise append one line to a log.
• When a document is superseded, remove it from the index or mark it superseded and exclude that mark from normal search. Keep the file in an archive folder if retention says you must hold it.
• Review collections that change often, such as prices and procedures, on a schedule you can keep. Signed contracts usually change only when someone replaces the file.

2026-04-02  travel-policy        replaced  sha256:9c1e…  people-team-bot
2026-04-02  2019-travel-policy   removed   superseded by travel-policy

Step 14

One-page checklist

Why A short list is what you can actually run before you call a corpus ready. Print this section, or keep it in the collection README.