The Nexus Forums > General MW Mod Search Engine

Yacoby

Aug 12 2008, 01:33 PM

I have failed to get in touch with Dark0ne regarding this, so hopefully he will see this:

QUOTE

I was thinking of building a search engine for Morrowind Mods (and possibly later Oblivion mods), and was wondered if I could index the mods your site (The name, author etc)? Obviously, the program would only look at between 10 and 30 pages a day (or less/more, up to you), and it would keep up with the new submitted mods via the "Updates recently" page (It would only have to do this once a day, looking at the previous days entries).
I am also happy to do the indexing at a certain time of the day, or spread over the day, however you want. (Or not at all, but if this is the case, it would be good to know).
Although I doubt you, or your advertisers would even notice my program, I though it only nice to ask you.

LHammonds

Aug 12 2008, 06:39 PM

QUOTE(Yacoby @ Aug 12 2008, 08:33 AM)

I have failed to get in touch with Dark0ne regarding this, so hopefully he will see this:

QUOTE

I was thinking of building a search engine for Morrowind Mods (and possibly later Oblivion mods), and was wondered if I could index the mods your site (The name, author etc)? Obviously, the program would only look at between 10 and 30 pages a day (or less/more, up to you), and it would keep up with the new submitted mods via the "Updates recently" page (It would only have to do this once a day, looking at the previous days entries).
I am also happy to do the indexing at a certain time of the day, or spread over the day, however you want. (Or not at all, but if this is the case, it would be good to know).
Although I doubt you, or your advertisers would even notice my program, I though it only nice to ask you.

Well, I would think the 1st thing you would do is setup your application to read the "Updates Today" RSS feeds.

The RSS file will give you the mod title, game (Oblivion or Morrowind), category, download link and file ID.

You would only need to read this file once or possibly twice a day with very minimal impact to TESNexus.

RSS Feed: New Files Today
RSS Feed: File Updates Today

LHammonds

Yacoby

Aug 12 2008, 09:55 PM

QUOTE(LHammonds @ Aug 12 2008, 07:39 PM)

Well, I would think the 1st thing you would do is setup your application to read the "Updates Today" RSS feeds.

The RSS file will give you the mod title, game (Oblivion or Morrowind), category, download link and file ID.

You would only need to read this file once or possibly twice a day with very minimal impact to TESNexus.

RSS Feed: New Files Today
RSS Feed: File Updates Today

LHammonds

Heya

The problem with using the RSS feeds is that it is updates today, and I don't think it is in the last 24 hrs, which would lead to this error:
11:58 - Scan RSS
11:59 - User uploads mod
00:00 - RSS reset mods upload missed

Where as if I look at the previously uploaded entries once a day, I don't miss any mods

Thanks for the suggestion though.

LHammonds

Aug 13 2008, 03:15 AM

BEGIN TOTAL SPECULATION

Maybe a query subscription service then?

You call a page and pass the date as a parameter and it pulls all the mods uploaded that day in an XML format?

Example:

DayQuery.php?date=20081231

Returns and XML-formatted data set including the following data:

File ID
Game
Category
Mod Name
Author
Uploader
Version
Small Description
Date Uploaded
Date Updated

Maybe an additional query for further and updated information based on the info you already have collected (based on file ID)

Example:

ModQuery.php?fileid=8440

Returns the following data set:

File ID
Game
Category
Mod Name
Author
Uploader
Version
Small Description
Date Uploaded
Date Updated
Total # of Downloads
Total # of Views
Total # of Comments
Total # of Images
Total # of Files
Total # of Ratings
Average Rating

I guess it would depend on how much this service would be worth for Dark0ne to invest into it.

END TOTAL SPECULATION

Michlo

Aug 13 2008, 06:59 AM

QUOTE(LHammonds @ Aug 12 2008, 11:39 AM)

QUOTE(Yacoby @ Aug 12 2008, 08:33 AM)

I have failed to get in touch with Dark0ne regarding this, so hopefully he will see this:

QUOTE

I was thinking of building a search engine for Morrowind Mods (and possibly later Oblivion mods), and was wondered if I could index the mods your site (The name, author etc)? Obviously, the program would only look at between 10 and 30 pages a day (or less/more, up to you), and it would keep up with the new submitted mods via the "Updates recently" page (It would only have to do this once a day, looking at the previous days entries).
I am also happy to do the indexing at a certain time of the day, or spread over the day, however you want. (Or not at all, but if this is the case, it would be good to know).
Although I doubt you, or your advertisers would even notice my program, I though it only nice to ask you.

Well, I would think the 1st thing you would do is setup your application to read the "Updates Today" RSS feeds.

The RSS file will give you the mod title, game (Oblivion or Morrowind), category, download link and file ID.

You would only need to read this file once or possibly twice a day with very minimal impact to TESNexus.

RSS Feed: New Files Today
RSS Feed: File Updates Today

LHammonds

Bloody hell, how did I miss that? Thanks, mate.

I'm now using the RSS feed as well.

Cheers.

And with that, I'm going home so no more posts from me tonight (I hear the sighs of relief).

Dark0ne

Aug 13 2008, 11:59 AM

I'm often skeptical of these things; I've had quite a few people ask if they can crawl the site with bots to make search engines and such things but I'm wary of them. I'd need to know exactly what the script was doing and what sort of calls it would be making to the server.

Yacoby

Aug 13 2008, 01:02 PM

QUOTE(Dark0ne @ Aug 13 2008, 12:59 PM)

I'm often skeptical of these things; I've had quite a few people ask if they can crawl the site with bots to make search engines and such things but I'm wary of them. I'd need to know exactly what the script was doing and what sort of calls it would be making to the server.

Heya

Having taken a look at how the site works, this is the list of requests I want to make to your site

For indexing already uploaded mods: (obviously the loop would be done over several months, and only needs to be done once)

CODE

for every category{
    for every page in that category{
        Download page
        (http://www.tesnexus.com/downloads/cat.php?id=#&page=##&orderby=date&order=ASC)

        for every mod on that page{
            Download description page:
            http://www.tesnexus.com/downloads/file/description.php?id=376
        }
    }
}

For keeping the db updated. Run once a day

CODE

Download Recent updates page
(http://www.tesnexus.com/downloads/recent.php)
for every new mod{
    download mod page
    (http://www.tesnexus.com/downloads/file.php?id=###)

    download description page
    http://www.tesnexus.com/downloads/file/description.php?id=###
}

for every updated mod{
    if  updated description{
        download description page
        http://www.tesnexus.com/downloads/file/description.php?id=###
    }
}

The data on mods that I would keep are:

QUOTE

ID
Author
Date uploaded

Rating
Number of Ratings
Downloads
Date Rating/Downloads was retrieved

Description
Category
Size of Download

When/if the scripts written, I am happy to send you the scripts before I start running them

EDIT
Let me know if you are, in theory, happy with the above requests

Yacoby

Sep 9 2008, 04:18 PM

Just to let you all know, I have given my scripts to Dark0ne, so hopefully he should come back with some comments and/or some bandwidth/page request limits to stick to.