Designed by TeamTNT and the late Ty Halderman, the Boom source port gave a complete overhaul to the vanilla Doom engine, one that modern mappers desparately needed. All of Doom's infamous bugs and limitations were fixed. Many new features which fit nicely into the engine were added, including a better line types system, translucency, friction and wind types, and much more. To put it more generally, the "Boom Engine" can be thought of as "Vanilla Doom++", which, while it still for the most part plays almost exactly like Vanilla Doom, mappers are given more freedom to design maps with the Boom standard. A large number of maps have been developed which require Boom to run. Because of the attractive editing features provided by Boom, many popular source ports have adopted support for these features. This has led to the term "Boom-compatible engine": such WADs may run on many different source ports provided that the source port used supports the Boom editing extensions. Most do, because Boom-compatible maps might be the most popular kind of Doom map out in existence today.
Much like Vanilla Doom, there isn't much of a way to play the original Boom engine, due to the fact it's written for DOS - and there's not much reason to, because even though TeamTNT hasn't worked on the Boom engine for quite a while (the latest release was 20 years ago, in 1999), new ports have taken their place as upgrades of the Boom engine. Extending Boom's feature set is what lead to the creation of MBF.
Vanilla ain't cutting it anymore. Not for this. (Pictured: Stardate 20X6 MAP07)
MBF
Short for "Marine's Best Friend", MBF is a port that builds on Boom that introduces even more features. Features include but are not limited to: bug fixes, such as a fix for the blockmap limit (MBF includes an internal blockmap builder), a "beta" mode, in which MBF emulates the behaviour of beta versions of Doom, higher-resolution 640x400 screen mode, and a means to change the level's sky texture. Of course, the problem is this was also written for DOS, so there's not much way for the modern user to play it without a lot of effort. That finally brings us to a source port you can play on modern machines: PrBoom.
This mapset requires MBF compatibility to play. (Pictured: Valiant MAP30)
PrBoom (short for "Proff Boom", named after its creator who went under that alias) is, most relevantly, a Windows and MacOS X compatible source port that emulates all of Boom and MBF's features. It started mearly as a conversion of PrBoom to Windows, but then it also ended up absorbing MBF's features into it. In addition to this, it also comes packaged GLBoom, which is an OpenGL-based renderer (which is fancy talk for "rendered using some modern rendering techniques that make use of your hardware better").
It's an old source port (the last version being from 2008), but you can download it here. For some of you, that might be too old (especially Windows 10 users), so for modern Boom enthusiasts, I recommend PrBoom+, mentioned below.
I will let the creator of PrBoom+ speak on what it does:
"PrBoom+ is a Doom source port developed from the original PrBoom project.
"The target of the project is to extend the original port with features that are necessary or useful for me personally and to all those interested in my work. It is worth noting that all changes introduced by me in no way break PrBoom's compatibility with the original Doom/Doom2 engines, and it is possible to be confident this will never happen in the future since compatibility is as important a merit of PrBoom as it is to me."
You can download PrBoom+ here.