Continued fraction algorithm involves finding the greatest common factor of two numbers by expressing the larger number as a continued fraction in terms of the smaller number and then determining the last integer in the fraction. This integer is the greatest common factor.