How Amazon Bedrock helped the UK’s Governors for Schools generate meaningful insights

Governors for Schools is a charity operating across England and Wales to find and place volunteers on school and academy governing boards. Amazon Web Services (AWS) has worked with Governors for Schools by providing financial support, in addition to more than 100 AWS employees applying to take on school governance roles, providing much needed technical expertise in education.

Emma Harris, head of partnerships development at Governors for Schools, wanted to see if there was data showing the effect governors have in schools, and therefore quantify the impact of the work the organization does. Emma reached out to AWS to see if we could help process Ofsted reports at scale and establish whether there was a correlation between school performance and good governance.

In this post, we explain how we used Amazon Bedrock to process unstructured documents and generate meaningful insights.

Processing unstructured documents with Amazon Bedrock

In England, schools are inspected through a body called the Office for Standards in Education (Ofsted), and a report is created for every school. The reports contain a grading in several categories and descriptions of the operation of the school.

We sourced a little more than 1,000 Ofsted reports for secondary academies. The reports were taken from September 2022 onwards (after the COVID-19 pandemic). After spending some time reading the reports and testing approaches, we settled on using Amazon Bedrock with two prompts.

Solution overview

The text is first extracted from the Ofsted report PDF and then passed to Amazon Bedrock as part of a prompt. The first prompt extracts relevant sentences from the report, and the second evaluates the sentiment of those sentences before finally recording the results. For a deeper dive on using Amazon Bedrock for sentiment analysis, refer to Analyze customer reviews using Amazon Bedrock. Figure 1 shows the process flow, which follows these high-level steps:

After extracting the text from the PDF, extract key data such as school name, Ofsted grade, and relevant sentences
Perform sentiment analysis

Figure 1. Sentiment analysis process flow.

Sentiment analysis solution

The sentiment analysis solution used the following steps.

Extract key data

We used a prompt to extract the school name, any references to governors or trustees at the academy, and the overall school grading. This first prompt is shown in the following code example.

Read the following Ofsted report carefully. 
    <report>{text}</report>
    Find 
    # the name of the school being assessed
    # all mentions of the role of the governors or trustees at the school.
    # the single overall grading of the school 
    # <example>{{"schoolName" : "exampleSchool1", "grade" : "Outstanding", "sentences" : ["The  governors provide a stable base for the school to operate.", "The  school relies on a strong board to provide stability", "The head teacher  receives appropriate support from the board of governors"]}}</example>

    Reply only with the JSON object. with an array containing the full sentence where all references occur.

Perform sentiment analysis

We used a second prompt to evaluate the sentiment of the sentences as either positive, neutral or negative. The prompt asks for a preference towards neutral if there is not explicit positivity or negativity. This avoids situations where sentences such as “John Doe is the chair of governors” were being evaluated as positive sentiment. The second prompt used is shown in the following code example.

Read the array of sentences carefully. <array>{array}</array>.

    For each sentence, judge whether it is giving a positive view of the role of the governors at the school, a neutral view, or a negative view.

    Reply with an array of JSON objects containing the sentence and a score of 1-3 representing negative, neutral, or positive for each sentence in the array. The shape should be
    [{{"sentence":"<sentence_text>”,"score","<score>"}}]
    
    Lean towards a neutral score (i.e., 2) if the sentence is not explicitly negative or positive

We iterated through the reports held in an Amazon Simple Storage Service (Amazon S3) bucket calculating an “average sentiment” for the governance at an academy alongside the overall grading. Neutral sentences (such as statements of fact) were excluded from average sentiment calculation. Instead, only sentences such as “The trustees provide strong leadership and direction for the academy” or “The governors have not established a good educational framework” were used to avoid bias towards neutral in the results.

Do good governors improve Ofsted outcomes?

As any good data scientist will tell you, correlation is not causation! What we can say is that in the 1,000 academy reports we processed, there is a correlation between better governor sentence sentiment and better Ofsted outcomes.

Ofsted grading spans four grades: “Inadequate,” “Requires Improvement,” “Good,” and “Outstanding.” Figure 2 shows the average sentiment found in Ofsted reports for schools with each grade. The graph shows that a positive sentiment (3) will generally accompany schools graded as Good and Outstanding. Neutral to negative sentiment (1–2) will generally accompany schools graded Inadequate.

Figure 2. Average sentence sentiment (Y-axis) against Ofsted grade (X-axis) showing a correlation between higher sentiment and Good and Outstanding schools.

The overall distribution of the data is shown in figure 3. The graph shows that there are more data samples for Outstanding and Good schools, with schools graded Inadequate having a flatter distribution of governor sentiment.

Figure 3. Full dataset plot with sentiment (X-axis) and Ofsted grade (Y-axis) showing a correlation between higher sentiment and stronger Ofsted outcomes.

For the scientists and statisticians, it is worth noting that the correlation is statistically significant (p-value of 0.0008) and is a moderate positive correlation (r-value 0.37) driven by the limited differences observed in average governor sentence sentiment between Good and Outstanding schools.

Conclusion

AWS is proud to collaborate with public sector organizations such as Governors for Schools to serve the broader public interest and deliver societal change. This work has also shown how organizations can use the power of AWS technologies to provide insights into their data and help them make data-driven decisions.

If you are interested in learning more about how AWS can support your public sector initiatives, fill out the form on the Contact Us page, and one of our representatives will put you in touch with the right team.

AWS Public Sector Blog