Background: Competency in anesthesia traditionally has been determined subjectively in practice. Optimal training in procedural skills requires valid and reliable forms of assessment. The objective was to examine a procedure-specific clinical assessment tool for ultrasound-guided axillary brachial plexus block for inter-rater reliability and construct validity in a clinical setting.Method: This was a prospective observational study. Using prior work at our institution, the clinical assessment tool was developed consisting of a 63-point task-specific checklist and a global rating scale. The anesthesiologists were assigned to three groups based on prior experience of performing an axillary block: group 1 ('novices') < 10 procedures, group 2 ('intermediates') 50-80 procedures and group 3 ('experts') > 100 procedures. Each participant performed two consecutive blocks that were videotaped. Two blinded independent experts trained in the marking of the tool evaluated the videotapes.Results: There were five participants per group. The inter-rater reliability between assessors was 0.842 and 0.795 for the checklist and global rating scale, respectively. There was a consistent difference between (each pair of) the three groups in terms of both the checklist and global rating scale (P < 0.05). For the checklist, expert vs. intermediate group P = 0.023, expert vs. novice group P < 0.001 and intermediate vs. novice group P = 0.019. For the global rating scale, expert vs. intermediate group P < 0.001, expert vs. novice group gave P < 0.001 and intermediate vs. novice group P = 0.023.Conclusion: The objective of task-specific checklist and global rating scale are reliable and valid measures of axillary block performance between different levels of expertise.