Selected researchLearning from thrust corrections during docking

Local CBF-QP Projection Credit Assignment for Reinforcement Learning in Terminal-Phase Spacecraft Rendezvous and Docking

Author preprint · 2026

Shiqi Ren1,2,3, Pengcheng Wang1,2,3,*
Affiliations and correspondence
1Innovation Academy for Microsatellites, Chinese Academy of Sciences
2University of Chinese Academy of Sciences
3Key Laboratory for Satellite Digitalization Technology, Chinese Academy of Sciences

*Corresponding author

PFCR-DQP architecture. The upper panel shows the policy, safety layer, actuator and relative dynamics. The lower panel shows the local projection credit path added during training.
PFCR-DQP uses the safety filter’s active constraints to teach a docking policy which thrust directions need correction.

Motivation

As a spacecraft approaches its docking target, the permitted closing speed falls. A reinforcement-learning policy proposes a thrust command, and a control barrier function quadratic program (CBF-QP) can correct it before it reaches the actuator.

A successful docking can hide repeated disagreement between the policy and the filter. A scalar intervention penalty tells the policy that its command was corrected. The QP also identifies the constraint that caused the correction and the directions along which the policy can still act.

Read the full abstract

External control-barrier-function quadratic programs (CBF-QPs) modify reinforcement-learning commands before execution, but projection obscures which action directions create constraint conflict. We propose PFCR-DQP, a post-filter-consistent residual-learning method that uses local QP geometry for policy credit in terminal-phase spacecraft rendezvous and docking. The active set and multipliers of the operational solve define a local affine projection model. Its auxiliary loss penalizes the current policy’s residual along active constraint normals while retaining tangential freedom. Numerical regularity checks determine which reference samples contribute.

In a five-seed comparison with the barrier-aligned PFCR-RC-AW reference, the selected DQP policies reduce the steps at which the safety layer seizes more than half of the per-axis thrust authority from 35.94% to 5.12% and the median correction magnitude from 0.3307 to 0.0312 N, while the rate of any numerically discernible correction falls from 67.30% to 54.84%. The nominal-command deficit falls from 0.3922 to 0.1102 N and the axis-summed actuator-output impulse from 205.9 to 77.2 N·s. Protected docking success is 99.5% versus 97.6%. The paired success interval includes zero. Transfer without retraining yields 98.4% and 97.4% success in nonlinear Tudat-T0 and T1 propagation, respectively. Development curves show substantial late-training degradation, distinguishing the gains of selected policies from sustained training stability.

Our Framework

PFCR-DQP adds a directional training signal to the barrier-aligned PFCR-RC-AW policy. Both controllers use the same safety filter and braking backup.

  1. Record the operational QP solve. When the primary QP supplies the command, store the nominal and filtered thrust, the active constraint rows and their multipliers.
  2. Recover the local projection. Check the numerical optimality conditions and separate regular samples from active-set switches and near-degenerate solutions. On a regular sample, the active rows define the tangent and normal directions.
  3. Assign credit during the policy update. An auxiliary loss penalizes the current policy’s residual along the active normals. Tangential directions remain available for docking. At deployment, the controller runs the policy, safety layer and actuator.

Across 102.4 million PFCR-DQP training transitions, 66.78% passed the local regularity checks. Of those, 69.45% had at least one active constraint.

Regular and boundary samples, active ranks, Jacobian traces, and initial gradient diagnostics across five training seeds.
Supplementary Fig. S5. Availability and geometry of local projection credit across the five PFCR-DQP training runs.

Spacecraft Rendezvous

We trained three controllers using Clohessy–Wiltshire–Hill (CWH) relative dynamics. One checkpoint per method and training seed was selected on development episodes and then frozen. Each selected policy flew 200 holdout episodes, with initial conditions, actuator parameters and observation noise paired across methods.

How much thrust does the safety filter correct?

A correction above 0.5 N exceeds half the available thrust on one axis. These large overrides fell from 35.94% to 5.12% of control steps when PFCR-DQP replaced PFCR-RC-AW. The median correction fell from 0.3307 to 0.0312 N, and actuator-output impulse fell from 205.9 to 77.2 N·s.

Correction exceedance curves, per-seed comparisons at the 0.5 N threshold, and the magnitude distribution of corrected steps.
Fig. 4. Safety-layer override magnitude on the frozen CWH holdout. Counting every correction above 10−5 N gives 67.30% for PFCR-RC-AW and 54.84% for PFCR-DQP.

Approach trajectories and protected docking

Across the five training seeds, PFCR-DQP reached 99.5% protected docking success, compared with 97.6% for PFCR-RC-AW. The paired difference was +1.90 percentage points, with a descriptive 95% interval from −0.80 to +5.40 points. Velocity-envelope excursions were tracked separately.

Trajectory occupancy in docking-axis position and cross-track radius, and remaining range versus elapsed time, for all three controllers.
Supplementary Fig. S2. Approach geometry and range histories. All valid transitions from the 3,000 CWH holdout episodes are included.

Transfer to nonlinear orbital propagation

Without retraining, PFCR-DQP reached 98.4% protected docking success under Tudat’s nonlinear point-mass model (T0) and 97.4% under a model with orbital perturbations, thrust errors and mass depletion (T1). The controller continued to use CWH internally. The paired success intervals versus PFCR-RC-AW include zero at both fidelities.

Protected docking success by truth-model fidelity and the paired change from Tudat-T0 to Tudat-T1 for each controller.
Fig. 7. Each method and training seed was evaluated on 100 simulated episodes per fidelity.

Training stability and checkpoint selection

On the development set, PFCR-DQP reached 98.4% mean protected docking success at 4.096 million transitions. This fell to 9.6% at 20.480 million. All policies used in the holdout and transfer tests were selected between 3.072 and 5.120 million transitions.

Protected docking success, safety-layer modification and impulse at six development checkpoints. Success deteriorates late in training.
Fig. 2. Development performance across training. Points show individual seeds, diamonds show their means, and horizontal lines show their ranges.

BibTeX

@unpublished{ren2026pfcrdqp,
  author = {Ren, Shiqi and Wang, Pengcheng},
  title  = {Local {CBF-QP} Projection Credit Assignment for Reinforcement Learning in Terminal-Phase Spacecraft Rendezvous and Docking},
  year   = {2026},
  note   = {Author preprint}
}
Download .bib