Docs

Architecture Review Exam

Timebox: 90 minutes

Scenario: You are designing an Enterprise Knowledge Copilot for architecture review. It must answer with citations from internal standards, support approved tool calls, trace every interaction, and provide a production readiness summary.

Section 1: Layering

1. Draw the six major layers of the system and state the responsibility of each layer.
2. Explain why the model runtime should not own retrieval policy.
3. Explain why the agent/workflow layer should not own long-term experiment lineage.

Section 2: Runtime Decision

You have three candidate serving options: hosted API, vLLM, and llama.cpp.

1. Create a decision matrix with at least six criteria.
2. Pick one runtime for the first production release and justify the choice.
3. Define the metrics that would force a runtime change.

Section 3: RAG Data Contract

1. Define the required document, chunk, metadata, and query fields.
2. Explain how access control should be enforced.
3. Define a retrieval evaluation plan.

Section 4: LLMOps And Evaluation

1. Define the trace schema.
2. Define a promotion gate for prompt, retrieval, and model changes.
3. Explain how MLflow-style lineage differs from LLM trace observability.

Section 5: Security And Governance

1. Identify five security risks specific to this copilot.
2. Define the tool governance policy.
3. Define what must be logged for auditability.

Section 6: Production Readiness

1. Create a release checklist.
2. Define rollback behavior.
3. Define three failure rehearsal scenarios.

Scoring Rubric