搜索引擎程序Lucene

  Lucene open-source search software.

lucene.apache.org/

Apache Lucene is a high-performance, full-featured text search engine library written in Java.

Online Documentation

This README file only contains basic setup instructions. For more comprehensive documentation, visit:

• Latest Releases: https://lucene.apache.org/core/documentation.html
• Nightly: https://ci-builds.apache.org/job/Lucene/job/Lucene-Artifacts-main/javadoc/
• Build System Documentation: help/
• Developer Documentation: dev-docs/
• Migration Guide: lucene/MIGRATE.md

Building

Basic steps:

1. Install OpenJDK 17 (exactly this version).
2. Clone Lucene's git repository (or download the source distribution).
3. Run gradle launcher script (gradlew).

We'll assume that you know how to get and set up the JDK - if you don't, then we suggest starting at https://jdk.java.net/ and learning more about Java, before returning to this README.

See Contributing Guide for details.

Contributing

Bug fixes, improvements and new features are always welcome! Please review the Contributing to Lucene Guide for information on contributing.

Discussion and Support

• Users Mailing List
• Developers Mailing List
• Issue Tracker
• IRC: #lucene and #lucene-dev on freenode.net

from https://github.com/apache/lucene

------

Solr open-source search software.

solr.apache.org/

Solr

Apache Solr is an enterprise search platform written in Java and using Apache Lucene. Major features include full-text search, index replication and sharding, and result faceting and highlighting.

Online Documentation

This README file only contains basic setup instructions. For more comprehensive documentation, visit https://solr.apache.org/guide/solr

Building with Gradle

Firstly, you need to set up your development environment (OpenJDK 11 or greater).

We'll assume that you know how to get and set up the JDK - if you don't, then we suggest starting at https://jdk.java.net/ and learning more about Java, before returning to this README. Solr runs with Java 11 and later.

As of 9.0, Solr uses Gradle as the build system. Ant build support has been removed.

To build Solr, run (./ can be omitted on Windows):

./gradlew assemble

NOTE: DO NOT use gradle command that is already installed on your machine (unless you know what you'll do). The "gradle wrapper" (gradlew) does the job - downloads the correct version of it, setups necessary configurations.

The first time you run Gradle, it will create a file "gradle.properties" that contains machine-specific settings. Normally you can use this file as-is, but it can be modified if necessary.

The command above packages a full distribution of Solr server; the package can be located at:

solr/packaging/build/solr-*

Note that the gradle build does not create or copy binaries throughout the source repository so you need to switch to the packaging output folder above; the rest of the instructions below remain identical. The packaging directory is rewritten on each build.

For development, especially when you have created test indexes etc, use the ./gradlew dev task which will copy binaries to ./solr/packaging/build/dev but only overwrite the binaries which will preserve your test setup.

If you want to build the documentation, type ./gradlew -p solr documentation.

Running Solr

After building Solr, the server can be started using the bin/solr control scripts. Solr can be run in either standalone or clustered (SolrCloud mode).

To run Solr in standalone mode, run the following command from the solr/ directory:

bin/solr start

To run Solr in clustered mode, run the following command from the solr/ directory:

bin/solr start -c

The bin/solr control script allows heavy modification of the started Solr. Common options are described in some detail in solr/README.txt. For an exhaustive treatment of options, run bin/solr start -h from the solr/ directory.

Running Solr in Docker

You can run Solr in Docker via the official image.

To run Solr in a container and expose the Solr port, run:

docker run -p 8983:8983 solr

In order to start Solr in clustered mode, run the following.

docker run -p 8983:8983 solr solr-fg -c

For documentation on using the official docker builds, please refer to the DockerHub page.
Up to date documentation for running locally built images of this branch can be found in the local reference guide.

There is also a gradle task for building custom Solr images from your local checkout. These local images are built identically to the official image except for retrieving the Solr artifacts locally instead of from the official release. This can be useful for testing out local changes as well as creating custom images for yourself or your organization. The task will output the image name to use at the end of the build.

./gradlew docker

For more info on building an image, run:

./gradlew helpDocker

Docker images can also be built from the Solr binary distribution (i.e. solr-<version>.tgz). Please refer to the Solr Docker README for more information.

Running Solr on Kubernetes

Solr has official support for running on Kubernetes, in the official Docker image. Please refer to the Solr Operator home for details, tutorials and instructions.

Gradle build and IDE support

• IntelliJ - IntelliJ idea can import the project out of the box. Code formatting conventions should be manually adjusted.
• Eclipse - Not tested.
• Netbeans - Not tested.

Gradle build and tests

./gradlew assemble will build a runnable Solr as noted above.

./gradlew check will assemble Solr and run all validation tasks unit tests.

./gradlew help will print a list of help commands for high-level tasks. One of these is helpAnt that shows the gradle tasks corresponding to ant targets you may be familiar with.

Contributing

Please review the Contributing to Solr Guide for information on contributing.

Discussion and Support

• Mailing Lists
• Issue Tracker (JIRA)
• IRC: #solr and #solr-dev on libera.chat
• Slack

from https://github.com/apache/solr

 

(全文搜索服务器 Solr
 

Solr 是一个开源的搜索服务器。Solr 使用 Java 语言开发，主要基于 HTTP 和 Apache Lucene 实现。文档通过Http利用XML加到一个搜索集合中。查询该集合也是通过 http收到一个XML/JSON响应来实现。它的主要特性包括：高效、灵活的缓存功能，垂直搜索功能，高亮显示搜索结果，通过索引复制来提高可用性，提 供一套强大Data Schema来定义字段，类型和设置文本分析，提供基于Web的管理界面等。

官方下载:http://www.apache.org/dyn/closer.cgi/lucene/solr/ )

----------------------------------------------------------------------

 

 搜索引擎Nutch

Nutch 是一个开源的、Java 实现的搜索引擎。它提供了运行自己的搜索引擎所需的全部工具。是一个应用程序，可以以 Lucene 为基础实现搜索引擎应用。Nutch的创始人是Doug Cutting，他同时也是Lucene、Hadoop和Avro开源项目的创始人。

下载地址：http://www.apache.org/dyn/closer.cgi/nutch/
-----------

Apache Nutch is an extensible and scalable web crawler.

nutch.apache.org/

For the latest information about Nutch, please visit our website at:

https://nutch.apache.org/

and our wiki, at:

https://cwiki.apache.org/confluence/display/NUTCH/Home

To get started using Nutch read Tutorial:

https://cwiki.apache.org/confluence/display/NUTCH/NutchTutorial

Contributing

To contribute a patch, follow these instructions (note that installing Hub is not strictly required, but is recommended).

0. Download and install hub.github.com
1. File JIRA issue for your fix at https://issues.apache.org/jira/projects/NUTCH/issues
- you will get issue id NUTCH-xxx where xxx is the issue ID.
2. git clone https://github.com/apache/nutch.git
3. cd nutch
4. git checkout -b NUTCH-xxx
5. edit files (please try and include a test case if possible)
6. git status (make sure it shows what files you expected to edit)
7. Make sure that your code complies with the [Nutch codeformatting template](https://raw.githubusercontent.com/apache/nutch/master/eclipse-codeformat.xml), which is basially two space indents
8. git add <files>
9. git commit -m “fix for NUTCH-xxx contributed by <your username>”
10. git fork
11. git push -u <your git username> NUTCH-xxx
12. git pull-request

IDE setup

Eclipse

Generate Eclipse project files

ant eclipse

and follow the instructions in Importing existing projects.

You must configure the nutch-site.xml before running. Make sure, you've added http.agent.name and plugin.folders properties. The plugin.folders normally points to <project_root>/build/plugins.

Now create a Java Application Configuration, choose org.apache.nutch.crawl.Injector, add two paths as arguments. First one is the crawldb directory, second one is the URL directory where, the injector can read urls. Now run your configuration.

If we still see the No plugins found on paths of property plugin.folders="plugins", update the plugin.folders in the nutch-default.xml, this is a quick fix, but should not be used.

Intellij IDEA

First install the IvyIDEA Plugin. then run ant eclipse. This will create the necessary .classpath and .project files so that Intellij can import the project in the next step.

In Intellij IDEA, select File > New > Project from Existing Sources. Select the nutch home directory and click "Open".

On the "Import Project" screen select the "Import project from external model" radio button and select "Eclipse". Click "Create". On the next screen the "Eclipse projects directory" should be already set to the nutch folder. Leave the "Create module files near .classpath files" radio button selected. Click "Next" on the next screens. On the project SDK screen select Java 11 and click "Create".

Once the project is imported, you will see a popup saying "Ant build scripts found", "Frameworks detected - IvyIDEA Framework detected". Click "Import". If you don't get the pop-up, I'd suggest going through the steps again as this happens from time to time. There is another Ant popup that asks you to configure the project. Do NOT click "Configure".

To import the code-style, Go to Intellij IDEA > Preferences > Editor > Code Style > Java.

For the Scheme dropdown select "Project". Click the gear icon and select "Import Scheme" > "Eclipse XML file".

Select the eclipse-format.xml file and click "Open". On next screen check the "Current Scheme" checkbox and hit OK.

Running in Intellij IDEA

Running in Intellij

• Open Run/Debug Configurations
• Select "+" to create a new configuration and select "Application"
• For "Main Class" enter a class with a main function (e.g. org.apache.nutch.indexer.IndexingJob).
• For "Program Arguments" add the arguments needed for the class. You can get these by running the crawl executable for your job. Use full-qualified paths. (e.g. /Users/kamil/workspace/external/nutch/crawl/crawldb /Users/kamil/workspace/external/nutch/crawl/segments/20221222160141 -deleteGone)
• For "Working Directory" enter "/Users/kamil/workspace/external/nutch/runtime/local".
• Select "Modify options" > "Modify Classpath" and add the config directory belonging to the "Working Directory" from the previous step (e.g. /Users/kamil/workspace/external/nutch/runtime/local/conf). This will allow the resource loader to load that configuration.
• Select "Modify options" > "Add VM Options". Add the VM options needed. You can get these by running the crawl executable for your job (e.g. -Xmx4096m -Dhadoop.log.dir=/Users/kamil/workspace/external/nutch/runtime/local/logs -Dhadoop.log.file=hadoop.log -Dmapreduce.job.reduces=2 -Dmapreduce.reduce.speculative=false -Dmapreduce.map.speculative=false -Dmapreduce.map.output.compress=true)

Note: You will need to manually trigger a build through ANT to get latest updated changes when running. This is because the ant build system is separate from the Intellij one.

from https://github.com/apache/nutch