Web-Bot Technologies – Preparing the Data

Web-Bot Technologies – Preparing the Data

Using a web crawler to index every bit of information ;
1) Divide everything into file types
2) Using MetaData, index keywords, sentences, paragraphs
3) You need to link keywords directly to the info
4) If your program does not understand keywords, sentences and paragraphs in machine language, you have failed, every bit of info can be understood in machine language eg email, tweet, link etc. A furthur enhancement you can search for the languages of the world but it is too complex to discuss here, cause we do not have the technologies now.
5) You need to write an algorithm to spur hundreds of webcrawlers to increase speed
6) Remember after indexing all of your info you need a huge cache to link to searches, once a keyword search is used, it will be kept in memory
7) So in fact everyone is searching for the info from your snapshot instead of directly from Internet
8) So frequency of updates is very important eg every 1 hour
9) Every piece of info is timestamped to give the user a choice to chose the most latest info.

– Contributed by Oogle. 

Author: Gilbert Tan TS

IT expert with more than 20 years experience in Multiple OS, Security, Data & Internet , Interests include AI and Big Data, Internet and multimedia. An experienced Real Estate agent, Insurance agent, and a Futures trader. I am capable of finding any answers in the world you want as long as there are reports available online for me to do my own research to bring you closest to all the unsolved mysteries in this world, because I can find all the paths to the Truth, and what the Future holds. All I need is to observe, test and probe to research on anything I want, what you need to do will take months to achieve, all I need is a few hours.​

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s