Trend Micro Inc. Interview Question

What is this algorithm? How to implement it in detail(Seriously very detailed)?