Objective To study the characteristics of stress distributions on a customized lingual self-locking orthodontic appliance under transient occlusal force and optimize its structure. Methods A whole 3D model including denture, appliances and wire was established by CT scanning, reverse engineering method and CAD technology; transient nonlinear dynamic analysis on this model during occluding and its structural optimization were conducted, and the optimized lingual appliance was made based on rapid prototyping technology to verify reliability of the finite element model. Results The equivalent stress on the bracket bottom was larger than that on other parts of the bracket; the maximum equivalent stress on the bracket cover was decreased by 60.9% after installing a reinforcing rib on it, which could effectively prevent stress concentration caused by the contact between the arch wire and bracket cover. The simulation results fundamentally agreed with the loading experiment on the bracket cover. Conclusions For lingual orthodontic treatment in clinic, the relative position between interaction points of the occlusal force and brackets should be concerned so as to avoid impairing the self-locking function; through optimizing the appliance design, the elastic potential energy of arch wire can be transferred more effectively to the teeth and reduce losses of the orthodontic force.