Audley

Yes, set it to 0. Turn it off completely. It's does NOT work properly. It will cause more problems than it solves.

In the .ini files for New Vegas change the following: 
 
iMultiSample=0 (this is AA)
iMaxAnisotropy=0 (this is AF)
 
They should be in your FalloutPrefs.ini file in MO under the Display section.

No, you're not missing something obvious. If you want to use ENBoost or ENB then hardware AA (i.e. MSAA) will not work in any way. Let me explain:
 
There are two common forms of Anti-Aliasing. The first is called hardware AA and things like Supersampling and Multi-Sampling (the games native AA) fall under this category. They apply AA before the scene is rendered. This puts a much heavier load on the GPU hence the name hardware AA as the GPU is doing all the heavy lifting. The other common form of AA is software AA. This form of AA is applied after the scene is rendered by the GPU; using complex algorithms it can detect aliased edges and blur them to look smooth. The good thing about software AA is that it puts much less load on the GPU and can, in most current games, look just as good as hardware AA. The bad thing is that in older games (especially ones with finicky engines) software AA is not nearly as effective.
 
So even though you have a good GPU, when using ENBoost or ENB hardware AA is disabled. This is because (correct me if I am wrong, please) when the d3d9.dll is injected into the game, it does so before the scene is rendered so it conflicts with the hardware AA. Not 100% sure why, but that is what I have come to learn through posting on forums and stuff. It's weird because the effects themselves are post-processed so they are rendered after the scene is loaded. I still don't fully understand it myself.
 
Anyway, Boris (creator of ENB and ENBoost) realizes this and he developed some some software AA in the ENB itself. But because software AA loads after the scene is rendered, it can be used alongside ENB. Another good thing about software AA is that you can combine different ones. Like enabling ENB Edge AA alongside SMAA. The bad thing is that because the game is so old, and the engine is so finicky, software AA does not look nearly as good as hardware AA.
 
Also, the reason the grass breaks is because it needs a form of hardware MSAA to look proper. This is known as transparency MSAA. That is a whole other discussion though.
 
So to summarize: if you wish to use ENB or ENBoost then you must deal with the aliasing. Because as far as I know you cannot use just ENBoost alongside hardware AA for Fallout 3 or New Vegas. Personally I hate aliasing and find it extremely ugly not to mention distracting. So I only use ENB rarely, if at all. 
 
I hope this explains it! :)

Ah I see what you are asking. 
 
Under the Display section: 
 
iMultiSample is Anti-Aliasing
 
iMaxAnisotropy is Anisotropic Filtering
 
And Under the Water section: 
 
bUseWaterDisplacements is Water Displacement
 
I hope this helps! :)

Is the Steam Overlay turned on? This causes all the mentioned issues with ENB. The two things are just not compatible. Other than that just double check you have hardware AA turned off. Check those two things and if the issue still persists we can go from there. 

With MO and New Vegas to use an ENB w/ the 4GB launcher you need to place the .d3d9 in the root folder and NOT the exes folder created by the 4GB loader. Even though everyone says otherwise. Though I don't know why considering that has never worked for me.