Try searching for "Foreskin's lament" or Sarah Silverman.

Or, say, "curse of adam and eve" which is from over a year ago, but should return first a thread of mine with the same name, but does not.

I have never found the google search to be overly helpful. It never had ALL of the threads, and was basically a matter of luck.
I assume this is because it is "spider" based and can only archive pages it actually sees, instead of the local search which actually searches the database.....

i think it used to work. actually i cannot find my "have we reversed the curse of adam and eve" thread on the internal search either.

A couple of comments:

1) The googlebot is a guest and cannot see any members only threads, so it will _never_ index any of "members only" topics.

2) I changed the robots.txt a few weeks ago, which may have caused some problems with the google indexing.

Is there a way to allow googlebot to see members only threads, so it can index them?

Uh yeah, but then anyone who searches will find them.....

yep.

It depends on why h.com was made members only in the first place.

1. If it was to encourage people to join, then enabling people to read through google like that won't hurt it because google's index has a delay of days (or more). That's not convenient. If anything, it will allow people to get a preview of what's inside the forum and encourage them to join.

2. If it was to hide information from unsuspecting eyes (i.e. from people who would normally not register on h.com) then perhaps it would be detrimental. But are we really that insecure?

I think you're confusing two issues:

1. That H.com was declared "Members Only" because of the server problems, and eliminating the guests enabled actual members to be able to access the site. It was not a "policy move" but a practical one (which I think has been reversed).

2. Certain forums remain members only like Members Corner, BYLR, AC and GQ. Those are NOT for google's eyes....

Oh, I didn't realize that was the case, and it must have been done before I stepped in to help. I guess Nechama or Silent J can switch the setting back then and those forums will be indexed.

This is still VERY broken

Did you just turn if off, and all I need to do is Enable -> yes?

No, what I did is on the webserver level.

What you need to do is make all the public forums available to guests. Currently some of them (like member corner) are not available to guests.

From my end it looks like all public forums are visable to guests.

Missing:

Community-->Member Corner


Can you enable it for guests?

That is deliberate. Member corner was changed to be "members only" so you have to be signed in to read it as it deals with quasi-personal info......

K-Reb is right


From my end (not the tech mumbo jumbo files) the "Search Engine Spiders" in the CP...


Is set to No.

Is that correct?

I thought the whole point of Member Corner is that it's a private forum?

still not working...

I know, I still need someone to explain to me what I should be changing.....

Sorry, I didn't see this thread until now. I'll look into it again.

Nechama,

If the mods don't want to enable Member Corner as public, there's nothing you can do to allow google to index the Member Corner pages. The settings in the admin control panel are not going to help and should remain disabled.

Moshi,

Are there threads that are

1) not in member corner
AND
2) missing from the google index

Yes, absolutely. Very few threads come up in Google search results.

Try "reversed curse adam" for the same example as in the beginning of the thread.
Try "listerine" which should bring up a thread I had in halacha about kashrut of listerine. but doesn't.

As you can see from the non-google search.

Ok, I made some changes. Check back in a couple of days and let me know if there's still a problem.

thanks!

still not working fully.. "Why won't my wife let me pee against the fence when we have company for a barbecue" shows up in internal search, but not in google. Same with "Listerine".

Ok, I'll change it to allow google to index every link (even redundant links). But I can't do it now and it takes a couple of days for the index to be updated anyway. I hope this doesn't cause a performance problem.

I think it's working now. Please verify.

sorry.. the above-referenced search terms still not coming up.

Ok. There are some other options for helping google index the site, but I need to contact Silent J. It may take some time.

thanks!

still broken...

I created a sitemap (https://www.google.com/webmasters/tools/docs/en/protocol.html) that updates every hour and allowed googlebot access to every page on the site in robots.txt, but it does not seem to help. The problem is ostensibly on Google's side, but I can't be sure of it.
Maybe someone else has a suggestion.

well that's not encouraging...

Something is weird. I'm 99% sure I posted once about wanting ramekins... and the post doesn't come up in a search.