AgentCore の Gateway でエージェントの A/B テストを LLM によるオンライン評価で行う

2026-09-22 aws llm

AgentCore には Gateway の後ろに Runtime を置くことで A/B テストする仕組みがある。以前試した harness だと Runtime を直で呼ぶことができなかったのでまずは自前のコードで Runtime を立てる。オンライン評価を行う際に入力となるトレースを出力するため X-Ray の権限が必要。

AgentCore harness でエージェントを動かし Code Interpreter によるサンドボックス環境でのコード実行や Gateway による外部ツールへの権限付与を試す - sambaiz-net

const runtimeRole = new iam.Role(this, 'RuntimeRole', {
  assumedBy: new iam.ServicePrincipal('bedrock-agentcore.amazonaws.com'),
  inlinePolicies: {
    agent: new iam.PolicyDocument({
      statements: [
        new iam.PolicyStatement({
          actions: ['bedrock:InvokeModel', 'bedrock:InvokeModelWithResponseStream'],
          resources: [
            `arn:aws:bedrock:*:${this.account}:inference-profile/*`,
            'arn:aws:bedrock:*::foundation-model/*',
          ],
        }),
        new iam.PolicyStatement({
          actions: ['bedrock-agentcore:GetConfigurationBundleVersion'],
          resources: [`arn:aws:bedrock-agentcore:${this.region}:${this.account}:configuration-bundle/*`],
        }),
        new iam.PolicyStatement({
          actions: [
            'logs:CreateLogGroup',
            'logs:CreateLogStream',
            'logs:PutLogEvents',
            'logs:DescribeLogStreams',
          ],
          resources: [`arn:aws:logs:${this.region}:${this.account}:log-group:/aws/bedrock-agentcore/*`],
        }),
        new iam.PolicyStatement({
          actions: ['xray:PutTraceSegments', 'xray:PutSpans', 'xray:PutSpansForIndexing'],
          resources: ['*'],
        }),
        new iam.PolicyStatement({
          actions: ['cloudwatch:PutMetricData'],
          resources: ['*'],
          conditions: { StringEquals: { 'cloudwatch:namespace': 'bedrock-agentcore' } },
        }),
      ],
    }),
  },
})

const runtime = new agentcore.Runtime(this, 'Runtime', {
  runtimeName: 'demo_ab_agent',
  agentRuntimeArtifact: agentcore.AgentRuntimeArtifact.fromAsset(
    path.join(__dirname, '..', '..', 'agent'),
  ),
  executionRole: runtimeRole,
  environmentVariables: { MODEL_ID: 'jp.anthropic.claude-sonnet-4-6' },
  tracingEnabled: true,
})

A/B テストの対象となる設定バンドルの内容は BedrockAgentCoreContext.get_config_bundle() で取得できる。

import logging
import os

from bedrock_agentcore.runtime import BedrockAgentCoreApp, BedrockAgentCoreContext
from strands import Agent
from strands.models import BedrockModel

logging.basicConfig(level=logging.INFO)
log = logging.getLogger("agent")

MODEL_ID = os.environ.get("MODEL_ID")
REGION = os.environ.get("AWS_REGION", "ap-northeast-1")

app = BedrockAgentCoreApp()
model = BedrockModel(model_id=MODEL_ID, region_name=REGION)

@app.entrypoint
def invoke(payload: dict) -> dict:
    prompt = payload.get("prompt")
    if not prompt:
        return {"error": "payload に prompt がありません"}
    
    bundle = BedrockAgentCoreContext.get_config_bundle()
    system_prompt = bundle.get("systemPrompt")
    agent = Agent(model=model, system_prompt=system_prompt)
    return {"output": str(agent(prompt))}


if __name__ == "__main__":
    app.run()

Gateway を立てて Target に Runtime を指定する。

const gatewayRole = new iam.Role(this, 'GatewayRole', {
  assumedBy: new iam.ServicePrincipal('bedrock-agentcore.amazonaws.com'),
  inlinePolicies: {
    forward: new iam.PolicyDocument({
      statements: [
        new iam.PolicyStatement({
          actions: ['bedrock-agentcore:InvokeAgentRuntime'],
          resources: [runtime.agentRuntimeArn, `${runtime.agentRuntimeArn}/*`],
        }),
        new iam.PolicyStatement({
          actions: ['bedrock-agentcore:GetConfigurationBundleVersion'],
          resources: [`arn:aws:bedrock-agentcore:${this.region}:${this.account}:configuration-bundle/*`],
        }),
      ],
    }),
  },
})

const gateway = new agentcore.CfnGateway(this, 'Gateway', {
  name: 'agentcore-ab-gateway',
  roleArn: gatewayRole.roleArn,
  authorizerType: 'AWS_IAM',
});

const target = new agentcore.CfnGatewayTarget(this, 'RuntimeTarget', {
  gatewayIdentifier: gateway.attrGatewayIdentifier,
  name: 'agent',
  credentialProviderConfigurations: [{ credentialProviderType: 'GATEWAY_IAM_ROLE' }],
  targetConfiguration: {
    http: { agentcoreRuntime: { arn: runtime.agentRuntimeArn, qualifier: 'DEFAULT' } },
  },
})

A/B テスト実行用のロールを作成する。

const abTestRole = new iam.Role(this, 'AbTestRole', {
  assumedBy: new iam.ServicePrincipal('bedrock-agentcore.amazonaws.com', {
    conditions: {
      StringEquals: { 'aws:SourceAccount': this.account },
      ArnLike: { 'aws:SourceArn': `arn:aws:bedrock-agentcore:*:${this.account}:ab-test/*` },
    },
  }),
  inlinePolicies: {
    abtest: new iam.PolicyDocument({
      statements: [
        new iam.PolicyStatement({
          sid: 'AgentCoreResources',
          actions: [
            'bedrock-agentcore:GetGateway',
            'bedrock-agentcore:GetGatewayTarget',
            'bedrock-agentcore:ListGatewayTargets',
            'bedrock-agentcore:CreateGatewayRule',
            'bedrock-agentcore:UpdateGatewayRule',
            'bedrock-agentcore:GetGatewayRule',
            'bedrock-agentcore:DeleteGatewayRule',
            'bedrock-agentcore:ListGatewayRules',
            'bedrock-agentcore:GetOnlineEvaluationConfig',
            'bedrock-agentcore:GetEvaluator',
            'bedrock-agentcore:GetConfigurationBundle',
            'bedrock-agentcore:GetConfigurationBundleVersion',
            'bedrock-agentcore:ListConfigurationBundleVersions',
          ],
          resources: [`arn:aws:bedrock-agentcore:*:${this.account}:*`],
          conditions: { StringEquals: { 'aws:ResourceAccount': this.account } },
        }),
        new iam.PolicyStatement({
          sid: 'CloudWatchLogsDescribe',
          actions: ['logs:DescribeLogGroups'],
          resources: ['*'],
        }),
        new iam.PolicyStatement({
          sid: 'CloudWatchLogs',
          actions: [
            'logs:DescribeIndexPolicies',
            'logs:PutIndexPolicy',
            'logs:StartQuery',
            'logs:GetQueryResults',
            'logs:StopQuery',
            'logs:FilterLogEvents',
            'logs:GetLogEvents',
          ],
          resources: [
            `arn:aws:logs:*:${this.account}:log-group:/aws/bedrock-agentcore/evaluations/*`,
            `arn:aws:logs:*:${this.account}:log-group:aws/spans`,
            `arn:aws:logs:*:${this.account}:log-group:aws/spans:*`,
          ],
        }),
      ],
    }),
  },
})

Runtime に来たリクエストをサンプリングし LLM に投げてオンライン評価させる。

const onlineEval = new agentcore.OnlineEvaluationConfig(this, 'OnlineEval', {
  onlineEvaluationConfigName: 'demo_ab_online_eval',
  executionStatus: agentcore.ExecutionStatus.ENABLED,
  evaluators: [
    agentcore.EvaluatorSelector.builtin(agentcore.BuiltinEvaluator.GOAL_SUCCESS_RATE),
    agentcore.EvaluatorSelector.builtin(agentcore.BuiltinEvaluator.HELPFULNESS),
  ],
  dataSource: agentcore.DataSourceConfig.fromAgentRuntimeEndpoint(runtime),
  samplingPercentage: 100,
});

設定バンドルを作る。

const bundleFor = (id2: string, name: string, prompt: string, message: string) =>
  new agentcore.CfnConfigurationBundle(this, id2, {
    bundleName: name,
    commitMessage: message,
    components: new CfnJson(this, `${id2}Components`, {
      value: { [runtime.agentRuntimeArn]: { Configuration: { systemPrompt: prompt } } },
    }),
  })

const control = bundleFor('BundleControl', 'abPromptC', `
あなたは助手です。
回答は必ず 20 文字以内にしてください。
理由、背景、具体例は一切書かないでください。`, 'control')

const treatment = bundleFor('BundleTreatment', 'abPromptT1', `
あなたは助手です。質問に答えてください。
守ること:
- 結論を先に書く。
- 根拠を箇条書きで 2〜3 点添える。
- 分からないことは分からないと書く。推測で補わない。`, 'treatment')

これらを指定して A/B テストを開始する。

$ aws bedrock-agentcore create-ab-test \
    --name promptab \
    --gateway-arn "arn:aws:bedrock-agentcore:ap-northeast-1:524580158183:gateway/agentcore-ab-gateway-itadkjkcbo" \
    --role-arn "arn:aws:iam::524580158183:role/AgentCoreAb-AbTestRole2B002E73-M34OrWBzVTIG" \
    --evaluation-config '{"onlineEvaluationConfigArn":"arn:aws:bedrock-agentcore:ap-northeast-1:524580158183:online-evaluation-config/demo_ab_online_eval-AhqMTH9AxJ"}' \
    --enable-on-create \
    --variants '[
      {"name":"C","weight":50,"variantConfiguration":{"configurationBundle":{
        "bundleArn":"arn:aws:bedrock-agentcore:ap-northeast-1:524580158183:configuration-bundle/abPromptC-6KBJZh7PP4",
        "bundleVersion":"53e53a6d-ca64-444d-b906-906aaeb12275"}}},
      {"name":"T1","weight":50,"variantConfiguration":{"configurationBundle":{
        "bundleArn":"arn:aws:bedrock-agentcore:ap-northeast-1:524580158183:configuration-bundle/abPromptT1-VakgiG2E4U",
        "bundleVersion":"ad3ecbe4-44bf-4b09-bf1d-43274c21cd32"}}}
    ]'
{
    "abTestId": "promptab-845058f909",
    "abTestArn": "arn:aws:bedrock-agentcore:ap-northeast-1:524580158183:ab-test/promptab-845058f909",
    "name": "promptab",
    "status": "CREATING",
    "executionStatus": "NOT_STARTED",
    "createdAt": "2026-09-22T01:24:03.134000+09:00"
}

リクエストを送ると Control, Variant の評価値が表示される。