Objective: Taking the three-dimensional motion capture system (MoCap) as the gold standard, we develop a deep learning fusion model based on bi-lateral long short-term memory recurrent neural network (BiLSTM) and linear regression algorithm to reduce the system error of the Kinect sensor in lower limb kinematics measurement. Methods: We recruited ten healthy male college students for experiments. We simultaneously collected 3D dimensional coordinates of the reflective markers and the lower limb joint centers using a MoCap system and a Kinect V2 sensor, respectively. The joint angles of lower limbs are calculated using inverse kinematic model. We construct a dataset using the lower limb joint angles via the MoCap system as the target and the angles via the Kinect system as the input. We trained a BiLSTM network and a linear regression model for all lower limb angles. A leave-one subject-out cross-validation method is employed to study the performance of the models. The coefficient of multiple correlations (CMC) and root mean square error (RMSE) are used to investigate the similarity and the mean deviation between the joint angle waveforms via the MoCap and the Kinect system. Results: In comparison with the linear regression algorithm, the BiLSTM has better performance in refining lower limb kinematics due to its ability of dealing highly nonlinear regression problems. Our deep learning refined model significantly reduces the system error of Kinect. The mean RMSEs for all joint angles are mainly less than 10°, and the RMSEs of the hip joint are less than 5°. The joint angle waveforms present very good similarity with the golden standard with the CMCs of greater than 0.7 except for hip rotation angle. Conclusions: The deep learning fusion model based markerless gait analysis system developed in this study can accurately assess lower limb kinematics, joint mobility, walking functions, and has potential to be a cheap, easy-to-use alternative tool for clinical and home rehabilitation.