Query Text Index

Searches for items that match your specified natural language text, Boolean expressions, or fields.

The Query Text Index API searches for content in the IDOL OnDemand databases. Your query can include natural language text, keywords, and Boolean expressions. The API returns documents from a specified text index that matches your query expression.

Quick Start

All queries must contain the text parameter. IDOL OnDemand performs language processing to match terms as broadly as possible, including stemming (reducing plurals and verb forms of a word to the same stem) and transliteration (converting accented characters to non-accented forms).

A wide range of other optional parameters allow you to return a more specific results set, or to organize your results.

You can use quotation marks to enable exact phrase search. For example:

POST /1/api/[async|sync]/querytextindex/v1?text="Red+Panda" AND Elephant

Many other Boolean and Proximity operators are available on the text parameter, for more information see Boolean and Proximity Operators.

Search Result Ranking

IDOL indexes assign weights to terms based on their relevance to the dataset. The Text Tokenization API lets you examine what the weights are in the wikipedia index as a basis. As a simplified example, if every document in your index is about red things, then the term red is not as relevant in the following query as the term Panda.

POST /1/api/[async|sync]/querytextindex/v1?text=Red+Panda

IDOL ranks query results according to various factors, such as the weights of the terms in the index, frequency in the returned documents, and proximity of the stemmed terms. However it does not operate on a match all terms basis and actually operates better with more text rather than less. Documents return according to how closely they match the entire query.

{
  "documents": [
    {
      "reference": "http://feeds.eonline.com/~r/eonline/topstories/~3/vnqTU2tF04c/these-adorable-red-panda-twins-that-were-born-in-new-zealand-are-helping-save-their-species",
      "weight": 86.75,
      "links": [
        "RED",
        "PAND"
      ],
      "index": "news_eng",
      "title": "These Adorable Red Panda Twins That Were Born in New Zealand Are Helping Save Their Species!",
      "content": {}
    },
    {
      "reference": "http://en.wikipedia.org/wiki/Red Panda Adventures",
      "weight": 84.59,
      "links": [
        "RED",
        "PAND"
      ],
      "index": "wiki_eng",
      "title": "Red Panda Adventures",
      "content": {}
    },
    {
      "reference": "http://en.wikipedia.org/wiki/Red panda",
      "weight": 84.59,
      "links": [
        "RED",
        "PAND"
      ],
      "index": "wiki_eng",
      "title": "Red panda",
      "content": {}
    },
   ...

Specify the Index

IDOL OnDemand offers various public text indexes readily available for search, like wikipedia in various languages, or news sources. The indexes parameter allows you to restrict your results to these text indexes. For example:

POST /1/api/[async|sync]/querytextindex/v1?text="Red+Panda" AND Elephant&indexes=wiki_eng

You can also search any index that you have created using the Create Text Index API by specifying it in the same way.

POST /1/api/[async|sync]/querytextindex/v1?text="Red+Panda" AND Elephant&indexes=myindex

You can specify multiple indexes by setting multiple versions of the indexes parameter. For example:

POST /1/api/[async|sync]/querytextindex/v1?text="Red+Panda" AND Elephant&indexes=myindex&indexes=wiki_eng

Modify the Output

You can set the print parameter to all to return all the fields for each document.

POST /1/api/[async|sync]/querytextindex/v1?text="Red+Panda"&indexes=wiki_eng&print=all

The default value for print is fields, which allows you to specify the fields you want to print out in the print_fields parameter. For example:

POST /1/api/[async|sync]/querytextindex/v1?text="Red+Panda"&indexes=wiki_eng&print_fields=WIKIPEDIA_TYPE,DRECONTENT

This example returns the WIKIPEDIA_TYPE, and DRECONTENT fields for the result documents.

...
{
      "index": "wiki_eng",
      "title": "Red panda",
      "content": {
        "WIKIPEDIA_TYPE": "species",
        "DRECONTENT": "The red panda (Ailurus fulgens), also called lesser panda and red cat-bear, is a small arboreal mammal native to the eastern Himalayas and southwestern China that has been classified as Vulnerable by IUCN as its wild population is estimated at less than 10,000 mature individuals.
         ..." }
...

Control the Number of Results

By default, IDOL returns up to six documents for each search. You can use the max_results to choose the exact number of documents that you want to return. For example:

POST /1/api/[async|sync]/querytextindex/v1?text="Red+Panda"&database_match=wiki_eng&max_results=100

You can set the total_results parameter to true to return the total number of results that match the query, assuming no maximum restrictions. This option is most useful for pagination.

POST /1/api/[async|sync]/querytexindex/v1?text="Red+Panda"&database_match=wiki_eng&total_results=true

In this case, the totalhits tag returns in the results:

{
  "documents": [
...
  ],
  "totalhits": "214"
}

Field Matching and Operations

The fieldtext parameter allows for many operations, and capabilties vary based on the Index Field Types of the fields in your index. For more information on the operators and the formats of the fieldtext parameter, see FieldText Operators.

POST /1/api/[async|sync]/querytextindex/v1?text="Red+Panda"&database_match=wiki_eng&fieldtext=MATCH{species}:WIKIPEDIA_TYPE&print_fields=WIKIPEDIA_TYPE

This example uses the MATCH operator to search for pages that have the field WIKIPEDIA_TYPE with the value species.

{
  "documents": [
    {
      "reference": "http://en.wikipedia.org/wiki/Red panda",
      "weight": 86.73,
      "links": [
        "RED",
        "PAND"
      ],
      "index": "wiki_eng",
      "title": "Red panda",
      "content": {
        "WIKIPEDIA_TYPE": "species"
      }
    },
    {
      "reference": "http://en.wikipedia.org/wiki/Giant panda",
      "weight": 85.71,
      "links": [
        "RED",
        "PAND"
      ],
      "index": "wiki_eng",
      "title": "Giant panda",
      "content": {
        "WIKIPEDIA_TYPE": "species"
      }
    },

Summarization

The summary parameter enables the display of small summaries of the returned documents. For example:

POST /1/api/[async|sync]/querytextindex/v1?text="Red Panda"&summary=context

This example uses the context option to return summaries that contain the query terms. This option is very useful when dealing with large documents, and where you want to automatically generate informative snippets.

{
      "title": "Red panda",
      "summary": "\n\nThe red panda (Ailurus fulgens), also called lesser panda and red cat-bear, is a small arboreal mammal native to the eastern Himalayas and southwestern China that has been classified as Vulnerable by IUCN as its wild population is estimated at less than 10,000 mature individuals...
}

Highlight Query Terms

The highlight parameter allows you to automatically highlight the relevant pieces of information in the result text. For example:

POST /1/api/[async|sync]/querytextindex/v1?text="Red Panda"&summary=context&highlight=summary_terms

In this case we are using summary_terms to highlight our query terms in the summary.

{ 
     "title": "Red panda",
      "summary": "The <span class=\"highlight\">red panda</span> ...
}

Sort Results

By default, IDOL OnDemand sorts results by order of relevance. You can use the sort parameter to adjust the sort order. For example, when Date Fields are available, you can use the sort parameter to return the results in order of the date field value:

POST /1/api/[async|sync]/querytextindex/v1?text="Red+Panda"&database_match=news_eng&sort=date&print_fields=DREDATE

This example orders results by the value in the DREDATE field, with the most recent document first.

{
  "documents": [
    {
      "reference": "http://www.chinadaily.com.cn/world/2014-01/29/content_17265266.htm",
      "weight": 75.95,
      "links": [
        "RED",
        "PAND"
      ],
      "index": "news_eng",
      "title": "Rare red panda twins born in New Zealand zoo",
      "content": {
        "DREDATE": "1392445391"
      }
    },
    {
      "reference": "http://feeds.eonline.com/~r/eonline/topstories/~3/vnqTU2tF04c/these-adorable-red-panda-twins-that-were-born-in-new-zealand-are-helping-save-their-species",
      "weight": 88.28,
      "links": [
        "RED",
        "PAND"
      ],
      "index": "news_eng",
      "title": "These Adorable Red Panda Twins That Were Born in New Zealand Are Helping Save Their Species!",
      "content": {
        "DREDATE": "1391556895"
      }
    },
Synchronous
https://api.idolondemand.com/1/api/sync/querytextindex/v1
Asynchronous
https://api.idolondemand.com/1/api/async/querytextindex/v1
Authentication

This API requires an authentication token to be supplied in the following parameter:

ParameterDescription
apikeyThe API key to use to authenticate the API request.
Input Source

This API accepts a single input source that can be supplied using one of the following parameters:

ParameterDescription
textThe query text.
fileA file containing the query text. Multi part POST only.
referenceAn IDOL OnDemand reference obtained from either the Expand Container or Store Object API. The corresponding query text is passed to the API.
urlA publicly accessible HTTP URL from which the query text can be retrieved.
Parameters

In addition to the above input source, this API accepts the following parameters:

NameTypeDescription
field_text
string The fields that result documents must contain, and the conditions that these fields must meet for the documents to return as results. See FieldText Operators.
start
number
The number of the first document to display. Default value: 1
max_page_results
number
The maximum number of results to return for this query from the absolute number of results returned. If you have set the Start parameter, max_page_results sets the maximum number of results to return from the total results set.
absolute_max_results
number
The absolute maximum number of results to return for this query. Default value: 6
indexes
array The name of an IDOL OnDemand text index that you want to search for results. You can use the public datasets, or your own text indexes. See Public Text Indexes. Default value: wiki_eng
query_profile
string The name of the query profile that you want to apply.
print
enum <print> The types of fields and content to display in the results. Default value: fields
print_fields
string The names of fields to print in the results.
highlight
enum <highlight> The highlighting option to use for the result text. Default value: off
min_date
string The earliest creation date or time that a document can have to return as a result. See Parameter Date Formats.
max_date
string The latest creation date or time that a document can have to return as a result. See Parameter Date Formats.
min_score
number The minimum percentage relevance that results must have to the query to return.
sort
enum <sort> The criteria to use for the result display order. By default, results are displayed in order of relevance. Default value: relevance
total_results
boolean Set to true to return an estimate of the total number of result documents, and the total number of documents and document sections in the query databases.
start_tag
string The opening HTML tag to use to highlight a match. Default value: <span style="background-color: yellow">
end_tag
string The closing HTML tag to use to highlight a match. If omitted, this is generated automatically from the start_tag.
summary
enum <summary> The type of summary to create for result documents. Default value: off
All parameters for this API are optional.
Enumeration Types

This API's parameters use the enumerations described below:

print
The types of fields and content to display in the results.
all All fields.
all_sections All fields and all sections.
date Date fields.
fields Print fields listed in the print_fields parameter.
none Do not print content fields.
no_results Do not print results.
parametric Parametric fields.
reference Reference fields.
highlight
The highlighting option to use for the result text.
off No highlighting.
terms Terms that match the query text.
sentences Sentences that contain query terms.
sort
The criteria to use for the result display order. By default, results are displayed in order of relevance.
relevance Relevance order (most relevant first).
reverse_relevance Relevance order (least relevant first).
date Date order (most recent first).
reverse_date Date order (oldest first).
autn_rank Order by the standard relevance adjustment field.
off No sorting.
summary
The type of summary to create for result documents.
concept Concept Summary
Contains sentences that are typical of the result content. These sentences can be from different parts of the result document.
context Context Summary
Contains sentences that are typical of the result content, biased by the terms in the query text.
quick Quick Summary
The first few sentences of the document.
off No summary

This API returns a JSON response that is described by the model below. This single model is presented both as an easy to read abstract definition and as the formal JSON schema.

Model
This is an abstract definition of the response that describes each of the properties that might be returned.
Query Text Index Response {
documents (array[Documents]) The details of the returned documents.
}
Documents {
reference (string, optional) The reference string that identifies the result document.
weight (number, optional) The percentage relevance that the result document has to the original query.
links (array[string], optional) The terms from the query that match in the results document.
index (string, optional) The database that the result returned from.
title (string, optional) The title of the result document.
summary (string, optional) The summary of the results document.
}
Model Schema
This is a JSON schema that describes the syntax of the response. See json-schema.org for a complete reference.
{
    "type": "object",
    "properties": {
        "documents": {
            "type": "array",
            "items": {
                "type": "object",
                "properties": {
                    "reference": {
                        "type": "string"
                    },
                    "weight": {
                        "type": "number"
                    },
                    "links": {
                        "type": "array",
                        "items": {
                            "type": "string"
                        }
                    },
                    "index": {
                        "type": "string"
                    },
                    "title": {
                        "type": "string"
                    },
                    "summary": {
                        "type": "string"
                    }
                }
            }
        }
    },
    "required": [
        "documents"
    ]
}
https://api.idolondemand.com/1/api/sync/querytextindex/v1
/developer/api/api-example/1/api/sync/querytextindex/v1
Examples
See this API for yourself - select one of our examples below.
Keyword Search
Simple keyword search for 'government' on wikipedia
Search and Highlight
Return the content of eight documents and highlight
Search and Limit Results
Return 15 documents about fish
Input Source
ParameterValue
text
file
reference
url
Parameters
NameTypeValue
field_text
string
start
number
max_page_results
number
absolute_max_results
number
indexes
array
query_profile
string
print
enum
print_fields
string
highlight
enum
min_date
string
max_date
string
min_score
number
sort
enum
total_results
boolean (Default: False)
start_tag
string
end_tag
string
summary
enum

ASync – Response An error occurred making the API request
Response Code:
Response Body

	

Making API Request…
Checking result of job

To try this API with your own data and use it in your own applications, you need an API Key. You can create an API Key from your account page - API Keys.

Output Refresh An error occurred making the API request View the raw output View Input
Rendered RawHtml Response
Result Display
Response Code:
Response Body:

		
Make this call with curl
curl


If you would like to provide us with more information then please use the box below:

We will use your submission to help improve our product.