This paper proposes a unified framework capable of searching both the principal and minor components by utilizing the penalty function method. Analytical results show that the new criterion will exhibit a unique global maximum or minimum if the weights span the principal subspace or the minor subspace, depending upon the sign of the penalty factor. We derive a unified algorithm for adaptive extraction of both the principal and minor components by simply reversing the sign of the learning rate. The new algorithm provides an adaptation that significantly improves the learning performance. Simulation results demonstrate the effectiveness of the proposed algorithm when compared with the gradient-based fixed step-size algorithm.