Prompt for doc?

by ChanningDai - opened Jan 16

Jan 16

Hi, thanks for the release!

I noticed that there is a special prompt for the query ("Represent the query for retrieving supporting documents: "). Does the doc also require one?

ChanningDai

Jan 17

It's likely to be "Represent the document for retrieval: ". Got it from MangoDB API doc.

michael-guenther

Jan 19

•

edited Jan 19

I would also be interested in this but maybe, you are right, I tried to reproduce some of the spreadsheet results and with this this document prompt I get comparable results on most tasks besides FinQARetrieval where the score is 2.5% slower. Where is the document page that you found?

Query Prompt	Document Prompt	Average	AILACasedocs	AILAStatutes	ChatDoctorRetrieval	DS1000Retrieval	FinQARetrieval	FinanceBenchRetrieval	HC3FinanceRetrieval	LegalQuAD	LegalSummarization
"Represent the query for retrieving supporting documents: "	""	66.996	41.228	50.638	68.268	63.739	80.892	90.944	65.531	69.9	71.82
"Represent the query for retrieving supporting documents: "	"Represent the document for retrieval: "	66.67955556	41.444	48.585	67	63.937	82.485	90.882	63.089	70.411	72.283
Results from Spreadsheet		67.096	42.844	48.56	66.97	63.87	85.03	90.88	62.97	70.34	72.4

ChanningDai

Jan 20

@michael-guenther Here is the doc: https://www.mongodb.com/docs/api/doc/atlas-embedding-and-reranking-api/operation/operation-createcontextualizedembedding#operation-createcontextualizedembedding-body-application-json-input_type

hongliu9903

Voyage AI org Jan 20

Thanks @ChanningDai for finding the doc prompt and @michael-guenther for sharing the eval results!

hongliu9903 changed discussion status to closed Jan 23

Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.

Tap or paste here to upload images

· Sign up or log in to comment