Aims
Working with Code4Health and first year students from UCL Department of Computer Science, we aim to produce a measurable improvement in the standard of Trauma & Orthopaedic (T&O) training by gathering feedback by text from the trainees, then supplying weekly collated group feedback to those hospitals training them.

We aim to pilot this in T&O within the Northern Deanery. The impact of the feedback would be assessed scientifically in a randomised trial between hospitals, after gaining consent from trainees.
The expectation would be this software would eventually be used by other healthcare specialties including all those supported by Health Education England (160,000 students).

Outline of proposed method

• Include a specific group of trainees (e.g for the pilot, only trainees with a National Training Number (ST3 upwards) in the Health Education North East T&O)

• Weekly texts sent to trainees at end of working week

• Anonymous pooling of the response data

• Questions regarding training could be multiple and should be configurable but the default would be “Please rate your training this week 0-10, or * for “cannot comment” or # to “change details””. This question and the responses should be configurable in the database.

• Weekly feedback from trainees to be provided back to trainers at a trust level (to help maintain anonymity of trainees)

• The impact of this intervention would be measured scientifically. Trusts to be randomised as “cross-over trial”- 3/6 months on each arm (i.e. consultants receiving their feedback vs not). Exact details to be confirmed having sought statistician’s advice.

Functionality requirements:
Administrators pages (via a login)

The database will require administration to allow the project to be initiated and maintained.
This administrator would set the group from a pick list (https://hee.nhs.uk/hee-your-area). They would pick from a specialty list that we will supply; T&O for the pilot.
The administrator would then invite the identified group of trainees to register - and they would have the ability to add new trainees and remove old ones as time progresses. This would be in the form of a webpage link they are sent.
The administration pages will have a section to add/change the trainee list, and the day/time for the texts to go out (normally 17:01hrs every Friday).
The administration pages will have an engagement page with the following information, “In the last 4 weeks the following trainees have left feedback about the following hospitals:…”.
This will stop people giving feedback about other places that they are not currently working in. No raw data such as individual scores can be seen by the administration team, or the data analysis team.
The administration pages require a download page where the raw data is collated but this should not show the individual details of the trainee who gave the feedback. It should contain:

• Date/time of feedback request

• Date/time of feedback received (only count for analysis within 48 hours)

• Current trust

• Trust feedback of collated data (Y/N)

• Score 1-10 or * or #.

The admin pages will have a section to add/change the feedback list. This is the email addresses to receive the feedback (e.g. the T&O trainers and trainees in the North East), and the day/time for this feedback to go out (normally 07:59hrs every Monday). Trainees pages (no login but via a unique link)

Trainees would need to consent to the data above being collected and stored and to being involved in the study. There would be information to trainees regarding a clear explanation of mode of security of data, exactly what is done with the data, how it is kept anonymous, and how it is going to be fed back to the trainers. It would explain that trainees have the ability to exit the study whenever they wish. This section would read like a terms and conditions page.

The trainees would then enter their details:

• Name

• Current hospital trust

• Mobile phone number

Text messages

Weekly texts to trainees need to be generated, sent out, and the responses (via text message) need to be collated.
A text would look like this. “We currently have you listed as training at XX hospital: Please rate your training this week 0-10, or * for “cannot comment”, or # to “update details”. This question and the responses should be configurable in the database. The “update details” option should send a web link back to them allowing a trainee to update their name, hospital or phone number, or to withdraw from the study.
A further ‘reminder notification’ at 24 hours if no response received (time/number of reminders modifiable). A maximum of 1 response per trainee per week to be included in analysis.

Weekly feedback to trainers

Mean scores for the hospital will be sent out in a league table to predefined email addresses (the feedback list). This will be accompanied by some explanatory text. It will include mean scores for all Trusts that are randomised for feedback and have therefore received feedback that week.
To determine if this software leads to an improvement in training, we will randomise the trusts to receive weekly feedback, or not. Administration pages will need to ensure output of data runs alongside the randomisation schedule and ensure trusts receive feedback only at the correct time. At any specific time point a trust may be randomised to either receive feedback or not, and that allocation may change (a cross over trial).