WinnieProMaxAI GUIDE
Menu

LESSON

Finish Advanced: design and evaluate an AI knowledge assistant

Intermediate · Automate a repeated task

Advanced — design and evaluate AI workflows · Lesson 8 of 8

A bounded final project

Your project is a fictional assistant for a small training centre. It answers questions from an approved document collection and prepares unsent reply drafts. It may not promise discounts, modify registrations or send messages. This scope lets you demonstrate retrieval, context, tool boundaries and evaluation without making a real customer commitment.

Deliver four items: a source pack, a completed prototype conversation, a system diagram and a test report. You can implement the prototype using an accessible chat with supplied evidence. Building a deployed application is a separate implementation step; the Advanced programme's goal is to explain, design and evaluate the workflow.

Prerequisites are the preceding Advanced lessons and the Base final exercise. Keep the whole project synthetic. If you later use real organisational records, permissions and data handling must be reviewed for that environment.

Use this source pack

All records are fictional. K1, current, public: “Photography basics runs on Wednesdays at 18:00. Standard fee €40. Booking is confirmed only after staff approval.” K2, current, staff-only: “Staff may propose a discount, but the centre manager must approve it.” K3, superseded, public: “Photography basics runs on Tuesdays at 17:00.” K4, current, public: “The centre has no published information about parking.”

The simulated visitor can access public records only. The current valid answer about the class is Wednesday at 18:00 for a standard fee of €40. It must preserve the staff-confirmation condition. K2 cannot be revealed to this visitor just because it is useful to answering a discount question.

For a manual prototype, give the chat only the permitted current evidence. Separately document the application filtering that would be needed in a real system. Do not pretend that a model asked to ignore K2 is equivalent to a backend that never supplied K2.

Run the prototype and keep evidence

Write a task instruction in your own words: answer from the supplied evidence, cite record IDs, preserve missing information and prepare drafts only. Run three visitor questions: “When is the class and what does it cost?”, “Book me a place” and “Is parking free?”

Expected first reply: Wednesday at 18:00, standard fee €40, with K1. The booking reply may prepare a request but cannot claim confirmation. The parking reply says the available material does not establish parking availability or cost, citing K4. It should suggest asking staff rather than inventing a free car park.

Save actual outputs and mark each claim supported, unsupported or incomplete. If you could not run a chat, label your answers predicted and the practical step not yet completed. An honest unfinished practice record is more useful than presenting an authored sample as a measured model result.

Test the workflow, not just the reply

Draw the intended application: authenticated scope where applicable, current-source filtering, retrieval, model response, validation, human review and an optional approved action. Put actual operations behind tools with narrow effects. Use CA02 and CA03 if you need a deeper tool-cycle example.

Add four failure tests: only K3 is found; no source is found; a document contains an instruction to reveal staff material; a future approved save loses its receipt. The design should expose stale evidence, avoid fabricating an answer, treat document text as data and check save status before repeating a write.

Record outcome, correction time, latency and any usage cost for each actual trial. Do not combine predictions and measurements in one total. If a setting changes, rerun the same cases before concluding it improved the system. A real release would additionally need implementation and operational checks outside this learning project.

Completion and further study

You complete Advanced when your portfolio contains the four deliverables, the three prototype questions and four failure cases, with no unacknowledged unsupported claim. You must be able to explain why you chose context or retrieval rather than training, which operations require tools, how access is enforced and how cost and review affect usefulness.

If a check fails, correct the relevant stage and repeat the affected case. This is a defined programme endpoint, not a claim of mastery over all AI or certification. It prepares you to scope a project and continue into implementation with informed questions.

Choose your next depth from the linked original documentation: a larger RAG implementation, tool/API construction or model training. The separate Claude route adds provider-specific architecture practice. Return to these sources when implementing; a broad learning programme should not attempt to duplicate every provider reference page.

Sources

Read the original sources to explore the topic and check specific claims.

Claude: evaluation design ↗

Claude: tool-call implementation ↗

Claude Cookbook: RAG implementation ↗

Hugging Face: training implementation ↗

Go deeper when you are ready

The basic explanation is enough to continue. Choose a more detailed article if you want the mechanics; original documentation is linked in Sources.

← Back to articles