1

I wasn't sure how to ask the question. But basically, it's a textbook scenario. I'm working on a site that's article based, but the article information is stored in a database. Then the page is rendered with the information in the database based on the requested article id:

For example: http://www.mysite.com/articles/9851

I'm new to SEO, so I'm wondering how engines are able to crawl the contents of pages like this and/or what I need to do in order to ensure that it will be crawled.

So for instance, this site. All of the articles/posts on this site appear to live in a database somewhere. The URL has an ID which looks like it is used to tell the server which data to use to generate the page -- so the page doesn't actually exist somewhere, but it's template does. When I search google, I might find one of these posts based on the content of the post.

I understand that crawlers normally just find a page and follow it's links and follow its links' links and so on, but how does that work when the site is search based like this? Do have to create a page that randomly picks articles out of the database so that the crawler can see it or something?

Sinaesthetic
  • 302
  • 2
  • 12
  • [Sharing your research helps everyone](http://meta.programmers.stackexchange.com/questions/6559/why-is-research-important). Tell us what you've tried and why it didn’t meet your needs. This demonstrates that you’ve taken the time to try to help yourself, it saves us from reiterating obvious answers, and most of all it helps you get a more specific and relevant answer. Also see [ask] – gnat Aug 01 '14 at 05:47
  • 2
    Basically search engines will start at the top of your page and from there follow your link structure. They will repeat this once in a while (depending on various factors maybe about once a month, or more often if the decide your site is important). In addition you can provide a [site map](http://en.wikipedia.org/wiki/Site_map) that helps them to understand your sites structures (or reveal links that are not reachable directly). – thorsten müller Aug 01 '14 at 07:02
  • @gnat, the first thing I mention is the post is that I am not sure how to ask the question because I'm not even sure what I'm looking for. If I can't ask a question in a human read forum using plain english, how am I supposed to ask a search engine? But to try an address your comment, every article about SEO I came across regarding dynamically generated pages only returned tips for SEO for AJAX. This isn't ajax. – Sinaesthetic Aug 01 '14 at 16:08
  • @thorstenmüller Ok so it follows links. I think that much I could have assumed, but how would it crawl things that are not linked like I described in the post; pages that are generated based on the ID fed in the query string? So if an article had a bunch of content in it, how does the content get crawled when the only way to get to the page holding the content is through it's ID? – Sinaesthetic Aug 01 '14 at 16:09
  • @thorstenmüller I updated my post. This is a perfect example of me searching for the right thing to search for. "*In addition you can provide a site map that helps them to understand your sites structures (or reveal links that are not reachable directly)*" -- Maybe this is closer to what I was searching for. Could you check out my post update and let me know if this is what you're referring to? – Sinaesthetic Aug 01 '14 at 16:22
  • Not currently, no. Users find articles by searching. I had planned on creating a recent posts kind of thing of course. I see what you're getting at though. So how often do things like that typically get crawled? Does that depend on the popularity of the site? Because if the there are hundreds per day and recent posts only shows maybe the last 100 or so. Orr... I guess create a recent posts with a link that queries the next 100 and so on? I think I get it. – Sinaesthetic Aug 01 '14 at 17:02
  • 1
    As far as I know it's roughly about once per month for 'average' sites. But you will see that sites like Stack Exchange or Reddit are crawled much more. Links on these sites appear more or less instantly on Google. It depends on a lot of things, mostly how important Google thinks you are, how often they actually find something new, meta data in your headers, how fast your server responds to requests and and and... – thorsten müller Aug 01 '14 at 18:08
  • 1
    You can use a sitemap (others already describes some details) or some kind of page with "posting history" where you keep a paginated list of everything posted. Even if it's not very helpful to your users it would ensure that everything can be reached by simple links. – thorsten müller Aug 01 '14 at 18:11

2 Answers2

1

At its very simplest, search engines are just reading HTML. The web page at a certain URL is just an HTML file to a search engine, so it doesn't know a database query is involved. It only knows that the HTML file contains text and links -- which is an overly simplified explanation, but accurate enough for this scenario.

@RomanMik has the right idea. You need to review Google's documentation, but regardless of the search engine, the process is the same:

  1. Submit your site to the search engine
  2. Your site needs links -- lots of links, and links with descriptive text -- to the rest of the pages on your site
  3. A robots.txt file in the root of your domain can help tweak the search indexers, usually by telling them to ignore certain directories or file types
  4. Create fresh content and update the home page and section pages of your site to link to the new content

A search indexer will start at your home page, index it, then it will move on to child pages in your site based on the links it finds in the document it is currently indexing.

Forget about thinking of search engines and your site in terms of your database. It's all HTML to a search engine, and they need fresh content, descriptive content, and links to other pages. Do that and you've cleared the first major hurdle.

Greg Burghardt
  • 34,276
  • 8
  • 63
  • 114
0

Since Google is the largest Search Engine, and ALL sites want to be crawled by google, I suggest you start with google's webmaster tools page. (https://www.google.com/webmasters/). It is full of helpful resources.

To answer your question, you provide a robot.txt or a sitemap with all links that you want search engines to crawl and index. Here's a detailed answer about how to submit a sitemap to google. https://support.google.com/webmasters/answer/183669?hl=en

Roman Mik
  • 131
  • 9