It's not ridiculous.
Sgt. Pepper is not the Beatles' best work in my opinion. It's just their most celebrated. Half of it is pretty crappy compared to the great things they did before and after that album. Compare "Sexy Sadie" to 'When I'm 64" there's just no comparison. Likewise with the Beach Boys, they have many interesting albums. I personally prefer many of them over Pet Sounds, although I do understand the emotional importance that it carries. I think Friends would make a larger impression on a person. It's unlike most music, and will take most off-guard. Pet Sounds is like handing a person a massive stack of papers to look through, when the paper they need is at the very bottom. Whereas Friends is just one sheet of finely sliced paper with its message written clearly.
I completely agree with you that Sgt Pepper is overrated completely. I much prefer most of their other albums. But it's a valid point that I made--you don't hear Beatles fans complaining that fans start with Rubber Soul, Revolver and Sgt Pepper--I don't see why Beach Boys fans have to be sticklers about this.