Background Musculoskeletal conditions involving the shoulder are common and, because of the importance of the upper limb and hand in daily function, symptoms in this region are commonly associated with functional impairment in athletic and non-athletic populations. Deriving a definitive diagnosis as to the cause of shoulder symptoms is fraught with difficulty. Limitations have been recognised for imaging and for orthopaedic special tests. 1 solution is to partially base management on the response to tests aimed at reducing the severity of the patient's perception of symptoms. 1 (of many) such tests is the Shoulder Symptom Modification Procedure (SSMP). The reliability of this procedure is unknown. Methods 37 clinician participants independently watched the videos of 11 patient participants undergoing the SSMP and recorded each patient's response as improved (partially or completely), no change or worse. Inter-rater reliability was assessed by Krippendorff's ¿, which ranges from 0 to 1. Results Krippendorff's ¿ was found to range from 0.762 to 1.000, indicating moderate to substantial reliability. In addition, short (3-hour) and longer (1-day) durations of training were associated with similar levels of reliability across the techniques. Conclusions Deriving a definitive structural diagnosis for a person presenting with a musculoskeletal condition involving the shoulder is difficult. The findings of the present study suggest that the SSMP demonstrates a high level of reliability. More research is needed to better understand the relevance of such procedures. Trial registration number ISRCTN95412360.