Tutorial: Validation with AI

Introduction

This tutorial shows how matchbox’s AI supported FHIR-Resource Validation can be used.

Combining FHIR validation with AI technologies is an opportunity to make the process more efficient and more accessible for non-specialists. Validation results for complex resources can be overwhelming to understand, therefore having AI analyze the outcome messages can be an efficient way of finding and understanding mistakes in the implementation of the resource and getting recommendations on how to fix it.

Matchbox offers two methods of validating with support by AI technologies.

Analyze Outcomes with AI in matchbox

Getting Started

To use AI analysis in matchbox the following parameters must be set when setting up the matchbox-server application properties:


`matchbox.fhir.context.llm.provider`	The LLM provider used for the AI analysis of validation.
`matchbox.fhir.context.llm.modelName`	The LLM model used for the AI analysis of validation.
`matchbox.fhir.context.llm.apiKey`	Your API key for the desired LLM provider.

In addition to these parameters, the default behavior for the analysis can be decided by setting one of these parameters:


`matchbox.fhir.context.analyzeOutcomeWithAI`	Whether the validation outcome should be analyzed by a LLM or not. Requires the LLM parameters to be correctly set.
`matchbox.fhir.context.analyzeOutcomeWithAIOnError`	Whether the validation outcome should be analyzed by a LLM, when it includes error or fatal issues, or not. Requires the LLM parameters to be correctly set.

These parameters can be overridden at runtime during specific validations to adjust the behavior for specific use cases. Example configuration (API-Key must be replaced):

matchbox:
  fhir:
    context:
      onlyOneEngine: true
      txServer: http://tx.fhir.org
      analyzeOutcomeWithAIOnError: true
      llm:
        provider: openai
        modelName: gpt-4o-mini
        apiKey: sk-xxx

Validating Resources in matchbox

Having set up the matchbox server in this way resources can be validated as usual in matchbox (see Tutorial: validation). The validation results will now additionally include an analysis by the LLM.

In the matchbox UI it will be displayed as the top message returned by the validation: AI Analysis of the Operation Outcome

When using the APIs $validate operation the AI analysis will appear as an additional item in the issues list. The text is returned in markdown format for clearer readability when interpreting it.

Example:

{
      "extension": [
        {
          "url": "http://hl7.org/fhir/StructureDefinition/rendering-style",
          "valueString": "markdown"
        }
      ],
      "severity": "information",
      "code": "informational",
      "details": {
        "text": "AI Analyze of the Operation Outcome"
      },
      "diagnostics": "```markdown\nThe resource has issues labeled as 'error':\n\n1. **Issue**: The property name must be a JSON Array, not an Object (at Patient).\n   - **Severity**: error\n   - **Expression**: Patient.name\n\n### Guidance:\nThe `name` property of the Patient resource should be represented as an array of objects, even if there is only one name entry. \n\n### Example Fix:\nChange your resource to the following format:\n\n```json\n{\n\t\"resourceType\": \"Patient\",\n\t\"name\": [\n\t\t{\n\t\t\t\"family\": \"Test\"\n\t\t}\n\t]\n}\n```\n\nThis will ensure the `name` field adheres to the FHIR specification.\n```"
    }

Using an MCP-Client to perform validations

Matchbox provides an MCP-Server, that can be interpreted by MCP-Client Applications, such as Claude Desktop or VS Codes Github Copilot integration. This allows these applications to use the validation tool from matchbox and provide the user with clear information about the resource and interactive troubleshooting.

AI Analysis via MCP

When validating resources via MCP, the analysis of the validation is not performed by the LLM configured in matchbox. Instead, matchbox delegates the analysis back to the LLM of the initiating MCP client.

Depending if the MCP client supports Sampling, matchbox uses two methods: - With Sampling: Matchbox requests the AI analysis from the client LLM via an MCP sampling request during the tool call. - Without Sampling: Matchbox returns the analysis prompt together with the validations Operation Outcome as part of the tool response.

Activation

The AI analysis for MCP validation requests is only executed if: - The validation parameter analyzeOutcomeWithAI=true is provided OR - The property matchbox.fhir.context.analyzeOutcomeWithAI is set to true in the application.yaml (and not overwritten by the validation parameter).

In any case, the validation parameter analyzeOutcomeWithAI will get set to false before the validation in order to skip the analysis by the LLM configured in matchbox. The Sampling or Prompt Injection analysis will still be performed.

Getting Started

To enable MCP capabilities for matchbox add the following configuration to the application.yaml:

spring:
  ai:
    mcp:
      server:
        enabled: true

The MCP-Server can be reached at the endpoint http://<<your-url>>/matchbox(v3)/mcp/message

Setting up matchbox for Claude Desktop

For Claude Desktop users, follow the instructions for setting up remote MCP-Servers by Anthropic.

Free users can still use the above guide, but are limited to one MCP server at this time.

Example Claude Desktop: Claude Desktop MCP example

  {
      "mcpServers": {
        "matchbox": {
          "command": "npx",
          "args": [
            "mcp-remote@latest",
            "http://localhost:8080/matchboxv3/mcp/message"
          ]
        }
      }
    }

Setting up matchbox for VS Code GitHub Copilot

First, make sure the chat.mcp.enabled setting in VS Code is enabled.

To access the validation tool in the GitHub Copilot integration for VS Code find the settings.json file located here: %APPDATA%\Roaming\Code\User.

Add matchbox as MCP-Server as such:

{
    //...
    "mcp": {
        "servers": {
            "matchbox": {
                "url": "http://<<your-url>>/matchboxv3/mcp/message"
            }
        }
    },
    //...
}

Restart VS Code and within the Copilot integration select Agent mode.

VS Code Agent setting

Select the Tools button to view the available tools.

VS Code Tool selection

For further information view: https://code.visualstudio.com/docs/copilot/chat/mcp-servers.

Example GitHub Copilot in VS Code:

VS Code MCP example