TTYREC files are recordings of text-based sessions, much like screencasts are recordings of graphical sessions. The Roguelike Gallery and other roguelike sites create ttyrec recordings of games played online, so that others can watch them. The Archiver will search rlgallery.org's ttyrecs for you.
As you might expect, ttyrecs are not video files that can be played in video players. They contain mostly text, and you will need a ttyrec player to watch them.
The original ttyrec program comes with ttyplay.
Termrec includes termplay, which is easy to use and works on Windows.
IPBT is a newer player that allows you to rewind while playing.
A ttyrec produced on one kind of terminal will not play properly on a different kind of terminal. Most modern terminals are similar and should produce compatible ttyrecs, but you might come across a file that won't work.
Replaying also might not work properly if your terminal is a different size from the one that produced the ttyrec. If things are getting drawn in the wrong places, try making your terminal bigger.
Unfortunately, ttyrecs don't contain any metadata which could tell you what the proper size is. But most recent files from the Gallery and similar sites work around this limitation by putting some invisible text at the beginning. You can read it by opening the ttyrec with a hex editor or text editor. There will be a line starting with "Size:" followed by the terminal width and height.
Note: if you are using ipbt, you must also tell ipbt what size terminal to use. Check ipbt's documentation.