Pages Index (pages.json) Documentation

The Pages Index is a machine-readable catalogue of content published on The Sunil Abraham Project (TSAP). It is generated automatically from page front matter by a Python script and exported as a JSON file named pages.json. The official TSAP repository uses GitHub Actions to keep the published index synchronised automatically.

The purpose of the Pages Index is to provide a structured representation of TSAP content that can be consumed by external tools, bots, scripts, search systems, and future applications without requiring direct access to Jekyll internals or repository source files.

Background

As TSAP grew beyond one thousand pages, it became increasingly desirable to expose a structured list of published content.

Human readers can navigate the website through categories, internal links, search engines, and navigation menus. Software tools, however, require a structured source of information.

Several future use cases were identified:

• Telegram bot integration.
• Search and discovery tools.
• Research utilities.
• Content analysis.
• Statistics and reporting.
• External applications consuming TSAP metadata.

A machine-readable index became increasingly useful as the project expanded.

Why the Pages Index Was Created

Without a dedicated index, external tools would need to:

• Read Markdown files directly.
• Parse front matter.
• Understand Jekyll conventions.
• Reconstruct URLs.
• Determine which pages should be included.
• Handle future structural changes.

This creates unnecessary complexity and tightly couples external tools to repository internals.

The Pages Index was therefore created as a simple, portable, and machine-readable representation of published TSAP content.

Architecture

The Pages Index is generated from page front matter.

The script scans Markdown files throughout the repository and extracts selected metadata fields.

Only pages containing a created field are included.

This rule was chosen because:

• TSAP pages are expected to contain a created field.
• Utility pages and temporary files can be excluded naturally.
• The resulting index focuses on published content.

The process is:

Markdown Files
        ↓
Front Matter Extraction
        ↓
Metadata Selection
        ↓
pages.json Generation
        ↓
Publication

The generated file becomes a structured catalogue of TSAP content.

Script Location

The Pages Index is generated by:

scripts/generate_pages_json.py

The script scans Markdown files throughout the repository, extracts selected front matter metadata, and generates a machine-readable JSON index.

Automation

The official TSAP repository includes a GitHub Actions workflow (.github/workflows/pages.yml) that automates maintenance of the Pages Index.

Whenever relevant Markdown content or the generator script changes, the workflow executes scripts/generate_pages_json.py. If the generated pages.json differs from the repository version, the updated index is committed automatically.

Generated File

The output file is:

pages.json

It is written to the repository root and published automatically by GitHub Pages.

Published URL:

https://sunilabraham.in/pages.json

Included Metadata

Each indexed page may contain:

• title
• description
• created
• date
• source
• authors
• categories
• permalink

A typical entry looks like:

{
  "title": "Example Page",
  "description": "Example description",
  "created": "2026-06-08",
  "date": "2026-05-01",
  "source": "Example Source",
  "authors": ["Example Author"],
  "categories": ["Example Category"],
  "permalink": "https://sunilabraham.in/example-page/"
}

Installation Requirements

The script was designed to remain lightweight and portable.

Requirements:

• Python 3
• PyYAML

On Ubuntu:

sudo apt install python3-yaml

Alternatively, within a Python virtual environment:

pip install pyyaml

No database is required.

No Jekyll plugin is required.

The official TSAP repository uses a GitHub Actions workflow to automate generation and publication of pages.json. The Python script can still be run manually for local development or by anyone maintaining their own copy of the repository.

Running the Generator

Navigate to the root of the repository:

cd /path/to/your/repository

Example:

cd ~/Projects/sunilabraham

Run the generator:

python3 scripts/generate_pages_json.py

Typical output:

Created pages.json with 1126 pages.

First Successful Build

The first successful generation occurred on 8 June 2026.

Results:

Created pages.json with 1048 pages.

Generated file size:

576 KB

This demonstrated that a machine-readable index of the entire project could be generated efficiently while remaining small enough for rapid download.

Maintenance Workflow

The official TSAP repository now maintains the Pages Index automatically.

When relevant Markdown content is committed and pushed to the repository, GitHub Actions executes scripts/generate_pages_json.py. If the generated pages.json differs from the repository version, the updated index is committed automatically.

The generator may still be run manually when developing locally, testing changes, or maintaining a separate copy of the repository.

Current Uses

The Pages Index was originally created to support retrieval systems and machine-readable access to TSAP content.

Current uses include:

• Wayback Machine archiving.
• Telegram bot development.
• Metadata retrieval.
• Project statistics.
• Machine-readable access for external tools.

Future tools may consume the same index without requiring direct access to repository source files.

Advantages and Limitations

Advantages:

• Simple architecture.
• Human-readable source data.
• Machine-readable output.
• No database required.
• No build plugins required.
• Compatible with GitHub Pages.
• Lightweight and portable.

Limitations:

• Automatic regeneration currently applies only to the official GitHub repository workflow. Users maintaining local copies of the repository may still generate the index manually if required. When used with GitHub Pages’ “Deploy from branch” publishing model, the automated commit may also result in a cancelled Pages deployment before the final deployment completes successfully. This is a GitHub Pages behaviour rather than a limitation of the generation script itself (see this issue)
• Only pages containing a created field are included.
• Metadata quality depends upon front matter quality.

These limitations are considered acceptable given the project’s emphasis on simplicity and maintainability.

Future Improvements

Potential future enhancements include:

• Further refinement of the GitHub Actions workflow, including optimisation of workflow triggers and additional validation before publication.
• Additional metadata fields.
• Category-specific exports.
• Author-specific exports.
• Change tracking.
• Additional machine-readable indexes.

Any future development should continue to prioritise transparency, portability, and compatibility with GitHub Pages.

Development History

Development began on 8 June 2026.

The immediate goal was to create a machine-readable representation of TSAP content that could be consumed by external tools without requiring direct access to repository files.

The chosen approach was deliberately simple. Rather than introducing databases, search engines, build plugins, or external services, a standalone Python script was created to scan Markdown files and export selected front matter metadata into a single JSON file.

The first successful run generated:

Created pages.json with 1048 pages.

The resulting file was approximately 576 KB in size and was published at:

https://sunilabraham.in/pages.json

This established the first structured content index for the project.

Following the Version 2.1 development cycle in July 2026, the Pages Index generation process was automated using GitHub Actions, eliminating the need for routine manual regeneration and publication in the official TSAP repository.

📄 This page was created on 8 June 2026. You can view its history on GitHub, preview the fileTip: Press Alt+Shift+G, or inspect the underlying source. Last updated on 13 July 2026.