国内开发的相关程序

1、SQLET - 开放源码的中文搜索引擎  

官方网站 http://www.sqlet.com/

SQLET，是Search & Query &Link, 加后缀 let，表示小的，小型的意思.打算建立一个能搜上亿张网页的基于主题功能的中文搜索引擎.支持3种索引方式：MySql_table_Index， Lucene_Index，SQLET_Index.网页抓取可以保存在文件系统及数据库里。自带WebServer.

2、菲度垂直搜索引擎代码

菲度http://www.faydu.net 为一个垂直在线搜索的演示版，主要对国内一些购物站点进行搜索整理，

现在开源测试版本的代码，供大家讨论。下载说明：

1》因为本程序是在服务器上运行，是在多个处理器下运行的，个人电脑上请控制线程数量

2》包含一个data 的数据库 还原到sql server

3》收集完毕默认在bin目录有licene生成的反排的索引文件

4》下载地址：http://www.faydu.net/download/code.rar

开放日期：2006-4-18   来源:http://blog.csdn.net/faydu/archive/2006/04/18/667997.aspx
语言：VB.net(c#)

二>、中文分词程序代码

1、计算所汉语词法分析系统 ICTCLAS

中国科学院计算技术研究所在多年研究基础上，耗时一年研制出了基于多层隐马模型的汉语词法分析系统 ICTCLAS(Institute of Computing Technology, Chinese Lexical Analysis System)，该系统的功能有：中文分词；词性标注；未登录词识别。分词正确率高达97.58%(最近的973专家组评测结果)，基于角色标注的未登录词识别能取得高于90%召回率，其中中国人名的识别召回率接近98%，分词和词性标注处理速度为31.5KB/s。ICTCLAS 和计算所其他14项免费发布的成果被中外媒体广泛地报道，国内很多免费的中文分词模块都或多或少的参考过ICTCLAS的代码。

下载页面：http://www.nlp.org.cn/project/project.php?proj_id=6

由于 ICTCLAS 是由 C 语言写成的，现在主流的开发工具用起来不太方便，于是有一些热心的程序员把 ICTCLAS 改为 Java 和 C# 等其他语言。

（1）fenci，Java 的 ICTCLAS，下载页面：http://www.xml.org.cn/printpage.asp?BoardID=2&id=11502

（2）AutoSplit，另一个 Java 的 ICTCLAS，已经找不到下载页面，点击本地下载

（3）小叮咚中文分词，曾经有下载页面，现在找不到了。据作者介绍，从 ICTCLAS 中改进，有 Java，C# 和 C++ 三个版本，介绍页面：http://www.donews.net/accesine

2、海量智能分词研究版

海量智能计算技术研究中心为了使中文信息处理领域的研究者们能够共同分享海量智能中心的研究成果，共同提高中文信息处理水平，特此发布《海量智能分词研究版》，供专家、学者和爱好者进行研究。

下载页面：http://www.hylanda.com/cgi-bin/download/download.asp?id=8


3、其他

（1）CSW中文智能分词组件

运行环境：Windows NT、2000、XP 或更高，可以在 ASP，VB 等微软的开发语言中调用。

简介: CSW中文智能分词DLL组件，可将一段文本自动的按常规汉语词组进行拆分,并以指定方式进行分隔，且可对其拆分后的词组进行语义、词频标注。其广范应用于各行各业的信息资料检索、分析。

下载页面：http://www.vgoogle.net/

（2） C# 写的中文分词组件

据作者介绍，一个 DLL 文件，可以做中英文分词组件。完全C#托管代码编写，独立开发。

下载页面：http://www.rainsts.net/article.asp?id=48

三>、开源spider一览

spider是搜索引擎的必须模块.spider数据的结果直接影响到搜索引擎的评价指标.

第一个spider程序由MIT的Matthew K Gray操刀该程序的目的是为了统计互联网中主机的数目

Spier定义(关于Spider的定义,有广义和狭义两种).

    * 狭义:利用标准的http协议根据超链和web文档检索的方法遍历万维网信息空间的软件程序.
    * 广义:所有能利用http协议检索web文档的软件都称之为spider.

其中Protocol Gives Sites Way To Keep Out The 'Bots Jeremy Carl, Web Week, Volume 1, Issue 7, November 1995 是和spider息息相关的协议,大家有兴趣参考robotstxt.org.
Heritrix

Heritrix is the Internet Archive's open-source, extensible, web-scale, archival-quality web crawler project.

Heritrix (sometimes spelled heretrix, or misspelled or missaid as heratrix/heritix/ heretix/heratix) is an archaic word for heiress (woman who inherits). Since our crawler seeks to collect and preserve the digital artifacts of our culture for the benefit of future researchers and generations, this name seemed apt.

语言:JAVA, (下载地址)http://sourceforge.net/project/s ... up_id=73833&package_id=73980
WebLech URL Spider   

WebLech is a fully featured web site download/mirror tool in Java, which supports many features required to download websites and emulate standard web-browser behaviour as much as possible. WebLech is multithreaded and comes with a GUI console.

语言:JAVA, (下载地址)   http://sourceforge.net/project/showfiles.php?group_id=38170

JSpider

A Java implementation of a flexible and extensible web spider engine. Optional modules allow functionality to be added (searching dead links, testing the performance and scalability of a site, creating a sitemap, etc ..

语言:JAVA, (下载地址)http://sourceforge.net/project/showfiles.php?group_id=65617

WebSPHINX

WebSPHINX is a web crawler (robot, spider) Java class library, originally developed by Robert Miller of Carnegie Mellon University. Multithreaded, tollerant HTML parsing, URL filtering and page classification, pattern matching, mirroring, and more.

语言:JAVA, (下载地址)http://sourceforge.net/project/showfiles.php?group_id=48810

PySolitaire

PySolitaire is a fork of PySol Solitaire that runs correctly on Windows and has a nice clean installer. PySolitaire (Python Solitaire) is a collection of more than 300 solitaire and Mahjongg games like Klondike and Spider.

语言ython , (下载地址)http://sourceforge.net/project/showfiles.php?group_id=86107

The Spider Web Network Xoops Mod Team     

The Spider Web Network Xoops Module Team provides modules for the Xoops community written in the PHP coding language. We develop mods and or take existing php script and port it into the Xoops format. High quality mods is our goal.

语言:php , (下载地址)   http://sourceforge.net/projects/tswnmoddev

Fetchgals

A multi-threaded web spider that finds free porn thumbnail galleries by visiting a list of known TGPs (Thumbnail Gallery Posts). It optionally downloads the located pictures and movies. TGP list is included. Public domain perl script running on Linux.

语言:perl , (下载地址)http://sourceforge.net/project/showfiles.php?group_id=110338

Where Spider

The purpose of the Where Spider software is to provide a database system for storing URL addresses. The software is used for both ripping links and browsing them offline. The software uses a pure XML database which is easy to export and import.

语言:XML , (下载地址)http://sourceforge.net/project/showfiles.php?group_id=115931

Sperowider

Sperowider Website Archiving Suite is a set of Java applications, the primary purpose of which is to spider dynamic websites, and to create static distributable archives with a full text search index usable by an associated Java applet.

语言:Java , (下载地址)http://sourceforge.net/project/showfiles.php?group_id=90254

SpiderPy

SpiderPy is a web crawling spider program written in Python that allows users to collect files and search web sites through a configurable interface.

语言ython , (下载地址)http://sourceforge.net/project/showfiles.php?group_id=55531

Spidered Data Retrieval

Spider is a complete standalone Java application designed to easily integrate varied datasources. * XML driven framework * Scheduled pulling * Highly extensible * Provides hooks for custom post-processing and configuration

语言:Java , (下载地址)http://sourceforge.net/project/showfiles.php?group_id=90769

webloupe

WebLoupe is a java-based tool for analysis, interactive visualization (sitemap), and exploration of the information architecture and specific properties of local or publicly accessible websites. Based on web spider (or web crawler) technology.

语言:java , (下载地址)http://sourceforge.net/project/showfiles.php?group_id=121963

ASpider

Robust featureful multi-threaded CLI web spider using apache commons httpclient v3.0 written in java. ASpider downloads any files matching your given mime-types from a website. Tries to reg.exp. match emails by default, logging all results using log4j.

语言:java , (下载地址)http://sourceforge.net/project/showfiles.php?group_id=126578

larbin

Larbin is an HTTP Web crawler with an easy interface that runs under Linux. It can fetch more than 5 million pages a day on a standard PC (with a good network).

语言:C++, (下载地址)http://sourceforge.net/project/showfiles.php?group_id=42562