Upon entering this room, you will notice that the masks are creating shadows on the windows and the ground from the main room.
You want to be looking at these and noticing the gaps/holes being created with the light and shadows, for example on the right, you can see the eyes open on the right most shadow and the mouth open on the next one along, indicating that these are 2 and 1 (right most, then left).
The two shadows on the left show the left most with the eyes open and the frown line, then the next one along with all four possibilities for openings/gaps, making these shadows be 3 and 4 (left most, then right).
So when using the bow in the main room, you want to be taking down the masks in this order; starting with the third one along, moving to the fourth one along, then the first one, then the second one, as shown below.
This will open the cage ahead to gain you access to the Unsettling Mask, to progress with the story!