Amazon Bedrock Evaluations

Amazon Bedrock provides evaluation tools for you to accelerate adoption of generative AI applications. Evaluate, compare, and select the foundation model for your use case with Model Evaluation. Prepare your RAG applications for production that are built on Amazon Bedrock Knowledge Bases or your own custom RAG systems by evaluating the retrieve or retrieve and generate functions.

Use retrieve and generate evaluations to evaluate the end-to-end retrieval-augmented generation (RAG) capability of your application. Ensure the generated content is correct, complete, limits hallucinations, and adheres to responsible AI principles. Either evaluate the performance of a Bedrock Knowledge Base or bring your own inference responses from your custom RAG system. Simply select an LLM to use as a judge with your Amazon Bedrock Knowledge Bases or for your custom RAG outputs, upload your dataset, and select the metrics most important for your evaluation.

Use RAG retrieval evaluations to evaluate the storage and retrieval settings of your Amazon Bedrock Knowledge Bases or custom RAG system. Ensure the retrieved content is relevant and covers the entire user query. Simply select an LLM to use as a judge, choose a Bedrock Knowledge Base to evaluate or include your custom RAG system retrievals in your prompt dataset, and select your metrics.

Amazon Bedrock Model Evaluation allows you to use automatic and human evaluations to select FMs for a specific use case. Automatic (Programmatic) model evaluation uses curated and custom datasets and provides predefined metrics including accuracy, robustness, and toxicity. For subjective metrics, you can use Amazon Bedrock to set up a human evaluation workflow in a few quick steps. With human evaluations, you can bring your own datasets and define custom metrics, such as relevance, style, and alignment to brand voice. Human evaluation workflows can use your own employees as reviewers or you can engage a team managed by AWS to perform the human evaluation, where AWS hires skilled evaluators and manages the complete workflow on your behalf. You can also use an LLM-as-a-Judge to provide high quality evaluations on your dataset with metrics such as correctness, completeness, faithfulness (hallucination), as well as responsible AI metrics such as answer refusal and harmfulness. You can evaluate Bedrock models or any model anywhere by bringing your own inference responses in your input prompt dataset.

Use the compare feature in evaluations to see the results of any changes you made to your prompts, the models being evaluated, your custom RAG systems, or Bedrock Knowledge Bases.