The two blocks reach the bottom of the bowl at the same time with the same speed $u=U=\sqrt{2gh}$, as discussed in the comments. These are the initial speeds in the collision.

Momentum and kinetic energy are both conserved during the elastic collision. Instead of applying conservation of KE it is simpler to apply the **Law of Restitution** with a coefficient $e=1$. This implies that the relative velocity of separation equals the relative velocity of approach.

If the final speeds are $v, V$ for small and large blocks respectively, taking the +ve direction as left for the smaller block and right for the larger, then conservation of linear momentum and the law of restitution give $$(M-m)u=mv-MV$$ $$2u=v+V$$ From these we get $$v=\frac{3M-m}{M+m}u$$ $$V=\frac{3m-M}{m+M}u$$ Note that if $M \gt 3m$ then the larger block does not rebound ($V\lt 0$); it continues moving in its initial direction, chasing the smaller block. The smaller block always rebounds because $3M \gt m$ if $M \gt m$ hence $v \gt 0$ always.

The heights reached after the collision by the smaller and larger blocks are respectively $$h'=\frac{v^2}{2g}=(\frac{3M-m}{m+M})^2h$$ $$H'=\frac{V^2}{2g}=(\frac{3m-M}{M+m})^2h$$

You are making the solution harder than it needs to be. There are 3 steps :

1. Calculate the velocities at collision, which occurs at the bottom of the circle and is $\sqrt{2gh}$ for each .

2. Calculate the velocities after collision. Think of it as a linear collision.

3. The final heights are again given by $v^2=2gh$ but using the velocities after collision.

edited Oct 8, 2018 by sammy gerbil