SoftwareEngineering

A trial to use LLMs to document an application without access to the code base. The LLM had access to UI, network traffic, and database.

Reconnaissance:

1. AI as the user: AI agent with access to Playwright MCP server navigates the application. Capture user journey, screenshots, dialog descriptions, flowchart. Conclusion: not sufficient for a comprehensive overview.
2. Change data capture: 2nd pass, AI follows UI path and looks up database changes after each click. DB triggers were setup to create a log of every INSERT and UPDATE. Conclusion: powerful way to deconstruct queries from the outside.

Specification:

• Refine path discovery prompt, more explicitly find and traverse all possible user actions. Run multiple times.
• Converge versions, fix discrepancies, create a single consolidated map of the user flow.
• Capture network traffic by analyzing JSON payloads to reverse-engineer the backend API (Swagger doc). Conclusion: possible but difficult, not very useful.

Rebuild:

• generate backlog tasks from markdown, screenshots and DB logic descriptions
• Feed stories into Replit

Observations:

• AI-built prototype is a very effective way to validate and improve the specification.
• LLM struggled to create a good implementation plan.
• Replit ignored specifications about the datbaase.

Context engineering is building dynamic systems to provide the right information and tools in the right format such that the LLM can plausibly accomplish the task.

A common reason agentic systems don’t perform is they just don’t have the right context. Garbage in, garbage out.

The context passed to the model may be bad for a few reasons:

• There is just missing context that the model would need to make the right decision.
• The context us formatted poorly

Prompt engineering is a subset of context engineering.

Example of good context engineering

• Tool use: LLM can access the tool to get information
• Short term memory: Create a summary of a lengthy conversaion and use it in the future.
• Long term memory: Being able to fetch information from previous preferences.
• Prompt engineering: Clearly enumerate instructions.
• Retrieval: fetch information dynamically and insert it into the prompt before calling the LLM.