Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Reformat text-embeddings and models sections #354

Merged
merged 2 commits into from
Jan 14, 2025
Merged

Reformat text-embeddings and models sections #354

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
8362d5a
reformat text-embeddings and models sections
Jan 13, 2025
df6c8e4
Merge branch 'main' into reformat-text-embedding-section
trentfowlercohere Jan 14, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 4 additions & 3 deletions fern/pages/models/the-command-family-of-models/command-beta.mdx
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -53,7 +53,8 @@ Install the SDK, if you haven't already.
Then, set up the Cohere client.

```python PYTHON
import cohere
import cohere

co = cohere.Client(api_key)
```

Expand All @@ -66,8 +67,8 @@ message = "Write an introductory paragraph for a blog post about language models
### Generate text

```python PYTHON
response = co.chat(
model='command',
response = co.chat(
model="command",
message=message,
)

Expand Down
2 changes: 1 addition & 1 deletion fern/pages/models/the-command-family-of-models/command-r-plus.mdx
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -56,7 +56,7 @@ The model has been trained to respond in the language of the user. Here's an exa

```python PYTHON
co.chat(
message="Écris une description de produit pour une voiture électrique en 50 à 75 mots"
message="Écris une description de produit pour une voiture électrique en 50 à 75 mots"
)
```

Expand Down
2 changes: 1 addition & 1 deletion fern/pages/models/the-command-family-of-models/command-r.mdx
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -56,7 +56,7 @@ The model has been trained to respond in the language of the user. Here's an exa

```python PYTHON
co.chat(
message="Écris une description de produit pour une voiture électrique en 50 à 75 mots"
message="Écris une description de produit pour une voiture électrique en 50 à 75 mots"
)
```

Expand Down
67 changes: 34 additions & 33 deletions fern/pages/text-embeddings/embed-jobs.mdx
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -21,47 +21,43 @@ If you have a large collection of text rather than a single file, bulk embedding
```python PYTHON
# Request
co.bulk_embed(
model: "string",
url: "string"(url to gcp file)
file_id: "string" (id to uploaded file)
text_field: "string" (column to get text from)
truncate:"string" {LEFT:DEFAULT,RIGHT}
model="string",
trentfowlercohere marked this conversation as resolved.
Show resolved Hide resolved
url="string", # (url to gcp file)
file_id="string", # (id to uploaded file)
text_field="string", # (column to get text from)
truncate="string", # {LEFT:DEFAULT,RIGHT}
)
```

#### Response

```python PYTHON
{
job_id:"string"
}
{job_id: "string"}
```

### Retrieve A Bulk Embed

#### Request

```python PYTHON
co.get_bulk_embed(
job_id: "string"
)
co.get_bulk_embed(job_id="string")
```

#### Response

```python PYTHON
# Response of co.get_bulk_embed():
{
job_id: "string",
status: "string", (COMPLETE, QUEUED, FAILED or RUNNING)
created_at: "string", (timestamp)
updated_at: "string", (timestamp)
input_url: "string",
input_file_id: "string",
output_file_id: "string",
model: "string",
truncate: "string", (LEFT or RIGHT)
percent_complete: float
job_id: "string",
trentfowlercohere marked this conversation as resolved.
Show resolved Hide resolved
status: "string", # (COMPLETE, QUEUED, FAILED or RUNNING)
created_at: "string", # (timestamp)
updated_at: "string", # (timestamp)
input_url: "string",
input_file_id: "string",
output_file_id: "string",
model: "string",
truncate: "string", # (LEFT or RIGHT)
percent_complete: float,
}
```

Expand All @@ -77,18 +73,23 @@ co.list_bulk_embed()

```python PYTHON
# Response of co.list_bulk_embeds():
{bulk_embed : [{
job_id: "string",
status: "string", (COMPLETE, QUEUED, FAILED or RUNNING)
created_at: "string", (timestamp)
updated_at: "string", (timestamp)
input_url: "string",
input_file_id: "string",
output_file_id: "string",
model: "string",
truncate: "string", (LEFT or RIGHT)
percent_complete: float
}, ...]}
{
bulk_embed: [
{
job_id: "string",
status: "string", # (COMPLETE, QUEUED, FAILED or RUNNING)
created_at: "string", # (timestamp)
updated_at: "string", # (timestamp)
input_url: "string",
input_file_id: "string",
output_file_id: "string",
model: "string",
truncate: "string", # (LEFT or RIGHT)
percent_complete: float,
},
...,
]
}
```

## Next Steps
110 changes: 68 additions & 42 deletions fern/pages/text-embeddings/reranking/overview.mdx
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -27,6 +27,7 @@ In this example, the documents being passed in are a list of strings:

```python PYTHON
import cohere

co = cohere.Client(api_key="<YOUR API KEY>")

query = "What is the capital of the United States?"
Expand All @@ -35,8 +36,15 @@ docs = [
"The Commonwealth of the Northern Mariana Islands is a group of islands in the Pacific Ocean that are a political division controlled by the United States. Its capital is Saipan.",
"Charlotte Amalie is the capital and largest city of the United States Virgin Islands. It has about 20,000 people. The city is on the island of Saint Thomas.",
"Washington, D.C. (also known as simply Washington or D.C., and officially as the District of Columbia) is the capital of the United States. It is a federal district. The President of the USA and many major national government offices are in the territory. This makes it the political center of the United States of America.",
"Capital punishment has existed in the United States since before the United States was a country. As of 2017, capital punishment is legal in 30 of the 50 states. The federal government (including the United States military) also uses capital punishment."]
results = co.rerank(model="rerank-v3.5", query=query, documents=docs, top_n=5, return_documents=True)
"Capital punishment has existed in the United States since before the United States was a country. As of 2017, capital punishment is legal in 30 of the 50 states. The federal government (including the United States military) also uses capital punishment.",
]
results = co.rerank(
model="rerank-v3.5",
query=query,
documents=docs,
top_n=5,
return_documents=True,
)
```

**Response**
Expand Down Expand Up @@ -102,103 +110,121 @@ Alternatively, you can pass in a JSON object and specify the fields you'd like t
```python PYTHON
query = "What is the capital of the United States?"
docs = [
{"Title": "Facts about Carson City","Content": "Carson City is the capital city of the American state of Nevada. At the 2010 United States Census, Carson City had a population of 55,274."},
{"Title": "The Commonwealth of Northern Mariana Islands","Content": "The Commonwealth of the Northern Mariana Islands is a group of islands in the Pacific Ocean that are a political division controlled by the United States. Its capital is Saipan."},
{"Title": "The Capital of United States Virgin Islands","Content": "Charlotte Amalie is the capital and largest city of the United States Virgin Islands. It has about 20,000 people. The city is on the island of Saint Thomas."},
{"Title": "Washington D.C.","Content":"Washington, D.C. (also known as simply Washington or D.C., and officially as the District of Columbia) is the capital of the United States. It is a federal district. The President of the USA and many major national government offices are in the territory. This makes it the political center of the United States of America."},
{"Title": "Capital Punishment in the US","Content": "Capital punishment has existed in the United States since before the United States was a country. As of 2017, capital punishment is legal in 30 of the 50 states. The federal government (including the United States military) also uses capital punishment."}
]
results = co.rerank(model="rerank-v3.5", query=query, documents=docs, rank_fields=['Title','Content'],top_n=5, return_documents=True)

{
"Title": "Facts about Carson City",
"Content": "Carson City is the capital city of the American state of Nevada. At the 2010 United States Census, Carson City had a population of 55,274.",
},
{
"Title": "The Commonwealth of Northern Mariana Islands",
"Content": "The Commonwealth of the Northern Mariana Islands is a group of islands in the Pacific Ocean that are a political division controlled by the United States. Its capital is Saipan.",
},
{
"Title": "The Capital of United States Virgin Islands",
"Content": "Charlotte Amalie is the capital and largest city of the United States Virgin Islands. It has about 20,000 people. The city is on the island of Saint Thomas.",
},
{
"Title": "Washington D.C.",
"Content": "Washington, D.C. (also known as simply Washington or D.C., and officially as the District of Columbia) is the capital of the United States. It is a federal district. The President of the USA and many major national government offices are in the territory. This makes it the political center of the United States of America.",
},
{
"Title": "Capital Punishment in the US",
"Content": "Capital punishment has existed in the United States since before the United States was a country. As of 2017, capital punishment is legal in 30 of the 50 states. The federal government (including the United States military) also uses capital punishment.",
},
]
results = co.rerank(
model="rerank-v3.5",
query=query,
documents=docs,
rank_fields=["Title", "Content"],
top_n=5,
return_documents=True,
)
```

In the `docs` parameter, we are passing in a list of objects which have the key values: `['Title' ,'Content']`. As part of the Rerank call, we are specifying which keys to rank over, as well as the order in which the key value pairs should be considered.

```python PYTHON
{
id='e8f55f3f-d86e-47d7-9b24-7feb18286505',
RerankResponse(
trentfowlercohere marked this conversation as resolved.
Show resolved Hide resolved
id="e8f55f3f-d86e-47d7-9b24-7feb18286505",
results=[
RerankResponseResultsItem(
document=RerankResponseResultsItemDocument(
text=None,
Content=(
'Washington, D.C. (also known as simply Washington or D.C., and officially as the District of Columbia) '
'is the capital of the United States. It is a federal district. The President of the USA and many major '
'national government offices are in the territory. This makes it the political center of the United States of America.'
"Washington, D.C. (also known as simply Washington or D.C., and officially as the District of Columbia) "
"is the capital of the United States. It is a federal district. The President of the USA and many major "
"national government offices are in the territory. This makes it the political center of the United States of America."
),
Title='Washington D.C.'
Title="Washington D.C.",
),
index=3,
relevance_score=0.8914433
relevance_score=0.8914433,
),
RerankResponseResultsItem(
document=RerankResponseResultsItemDocument(
text=None,
Content=(
'Charlotte Amalie is the capital and largest city of the United States Virgin Islands. '
'It has about 20,000 people. The city is on the island of Saint Thomas.'
"Charlotte Amalie is the capital and largest city of the United States Virgin Islands. "
"It has about 20,000 people. The city is on the island of Saint Thomas."
),
Title='The Capital of United States Virgin Islands'
Title="The Capital of United States Virgin Islands",
),
index=2,
relevance_score=0.40344992
relevance_score=0.40344992,
),
RerankResponseResultsItem(
document=RerankResponseResultsItemDocument(
text=None,
Content=(
'Carson City is the capital city of the American state of Nevada. At the 2010 United States Census, '
'Carson City had a population of 55,274.'
"Carson City is the capital city of the American state of Nevada. At the 2010 United States Census, "
"Carson City had a population of 55,274."
),
Title='Facts about Carson City'
Title="Facts about Carson City",
),
index=0,
relevance_score=0.23343581
relevance_score=0.23343581,
),
RerankResponseResultsItem(
document=RerankResponseResultsItemDocument(
text=None,
Content=(
'The Commonwealth of the Northern Mariana Islands is a group of islands in the Pacific Ocean that '
'are a political division controlled by the United States. Its capital is Saipan.'
"The Commonwealth of the Northern Mariana Islands is a group of islands in the Pacific Ocean that "
"are a political division controlled by the United States. Its capital is Saipan."
),
Title='The Commonwealth of Northern Mariana Islands'
Title="The Commonwealth of Northern Mariana Islands",
),
index=1,
relevance_score=0.15964958
relevance_score=0.15964958,
),
RerankResponseResultsItem(
document=RerankResponseResultsItemDocument(
text=None,
Content=(
'Capital punishment has existed in the United States since before the United States was a country. '
'As of 2017, capital punishment is legal in 30 of the 50 states. The federal government (including the United States military) '
'also uses capital punishment.'
"Capital punishment has existed in the United States since before the United States was a country. "
"As of 2017, capital punishment is legal in 30 of the 50 states. The federal government (including the United States military) "
"also uses capital punishment."
),
Title='Capital Punishment in the US'
Title="Capital Punishment in the US",
),
index=4,
relevance_score=0.10465127
relevance_score=0.10465127,
),
],
meta=ApiMeta(
api_version=ApiMetaApiVersion(
version='1',
is_deprecated=None,
is_experimental=None
version="1", is_deprecated=None, is_experimental=None
),
billed_units=ApiMetaBilledUnits(
images=None,
input_tokens=None,
output_tokens=None,
search_units=1.0,
classifications=None
classifications=None,
),
tokens=None,
warnings=None
)
}

warnings=None,
),
)
```

## Multilingual Reranking
Expand Down
Loading