question: Here is a question from 2022666109.pdf document : Are Welsh ministers required to consult the natural resources body for Wales before making regulations?

data_source: DocBench

task_type: mmreasoning

modality: vl


question: Here is a question from NYSE_XOM_2020.pdf document : Did the average production costs for bitumen per barrel change from 2019 to 2020?

data_source: DocBench

task_type: mmreasoning

modality: vl


question: Here is a question from ASX_LOV_2020.pdf document : Was the net cash from operating activities higher in 2020 than in 2019?

data_source: DocBench

task_type: mmreasoning

modality: vl


question: Here is a question from 2020.emnlp-main.213.pdf document : Which language pair in the WMT 2018 DARR statistics has the highest average number of tokens in the source text?

gold_answers: The language pair en-zh (English-Chinese) has the highest average number of tokens in the source text with 28.27.

data_source: DocBench

task_type: mmreasoning

modality: vl


question: Here is a question from D18-1034.pdf document : Does the method described project word embeddings into a shared space for translation?

data_source: DocBench

task_type: mmreasoning

modality: vl


question: Here is a question from N19-1421.pdf document : What is the accuracy percentage of the BERT-LARGE model when answering questions about factoid knowledge according to Table 6?

data_source: DocBench

task_type: mmreasoning

modality: vl


question: Here is a question from Tab-2-INCSR-Vol-2-508.pdf document : What are some common sources of laundered funds in Trinidad and Tobago?

gold_answers: Illicit trafficking, fraud, tax evasion, corruption, and illegal gaming.

data_source: DocBench

task_type: mmreasoning

modality: vl


question: Here is a question from 2019713412.pdf document : Is plea bargaining allowed in Japan according to the Law Library of Congress document?

data_source: DocBench

task_type: mmreasoning

modality: vl


question: Here is a question from NASDAQ_AMGN_2020.pdf document : How much did total assets of the company change from 2018 to 2020?

gold_answers: Total assets decreased by $3,468 million from 2018 to 2020.

data_source: DocBench

task_type: mmreasoning

modality: vl


question: Here is a question from 20240222.pdf document : Which city is predicted to have the largest drop in office worker spending due to the shift to hybrid work?

data_source: DocBench

task_type: mmreasoning

modality: vl


question: Here is a question from 20230616.pdf document : Is Donald J. Trump's legal team planning to change the level of independence currently held by the Justice Department?

data_source: DocBench

task_type: mmreasoning

modality: vl


question: Here is a question from FBS_S_GWI_Public.pdf document : What is the primary focus of Bureau Objective 3.4?

gold_answers: The report does not contain such objective.

data_source: DocBench

task_type: mmreasoning

modality: vl


question: Here is a question from NYSE_UNH_2020.pdf document : By how much did the earnings from operations for UnitedHealthcare grow from 2018 to 2020?

gold_answers: The earnings from operations for UnitedHealthcare grew by $3,246 million from 2018 ($9,113 million) to 2020 ($12,359 million).

data_source: DocBench

task_type: mmreasoning

modality: vl


question: Here is a question from OTC_LVMUY_2021.pdf document : Which business group had the highest percentage increase in water consumption from 2020 to 2021?

gold_answers: The Wines & Spirits group had the highest percentage increase in water consumption, with a 23% increase.

data_source: DocBench

task_type: mmreasoning

modality: vl


question: Here is a question from P18-1009.pdf document : Does the presented data collection process prioritize precision or comprehensiveness?

data_source: DocBench

task_type: mmreasoning

modality: vl


question: Here is a question from 2020.findings-emnlp.139.pdf document : What is the total number of datapoints for Python when combining both bimodal and unimodal codes?

gold_answers: The total number of datapoints for Python is 1,614,304, which is the sum of bimodal datapoints (458,219) and unimodal codes (1,156,085).

data_source: DocBench

task_type: mmreasoning

modality: vl


question: Here is a question from 2022666113.pdf document : What approach has the EUAA recommended for conducting personal interviews in light of the COVID-19 pandemic?

gold_answers: The EUAA has published recommendations on how to conduct personal interviews remotely.

data_source: DocBench

task_type: mmreasoning

modality: vl


question: Here is a question from NYSE_PM_2020.pdf document : On which page does the report provide Mine Safety Disclosures?

data_source: DocBench

task_type: mmreasoning

modality: vl


question: Here is a question from 2021687414.pdf document : What is the section about Germany mainly talking about?

gold_answers: The document does not contain a section about Germany.

data_source: DocBench

task_type: mmreasoning

modality: vl


question: Here is a question from 2020.emnlp-main.213.pdf document : For the en-ru language pair, which COMET model performed better, the COMET-HETER Estimator or the MQM Estimator, and by how much did it outperform the other?

gold_answers: For the en-ru language pair, the COMET-MQM Estimator performed better than the COMET-HETER Estimator by a margin of 0.076 (COMET-MQM score of 0.615 minus COMET-HETER score of 0.539).

data_source: DocBench

task_type: mmreasoning

modality: vl


question: Here is a question from 20231213.pdf document : Did the polls by The New York Times and Siena College indicate a lead for the Democratic candidate in Pennsylvania's Senate race?

data_source: DocBench

task_type: mmreasoning

modality: vl


question: Here is a question from NYSE_PG_2022.pdf document : Which geographic region contributed the least to the P&G's net sales?

gold_answers: Both Latin America and India, Middle East & Africa (IMEA) contributed the least with 6% each.

data_source: DocBench

task_type: mmreasoning

modality: vl


question: Here is a question from D18-1034.pdf document : Which model achieved the highest F1 score on the Uyghur Unsequestered Set according to Table 3?

gold_answers: The "Combined + self-att." model achieved the highest F1 score on the Uyghur Unsequestered Set with a score of 32.09 ± 0.61.

data_source: DocBench

task_type: mmreasoning

modality: vl


question: Here is a question from 2021687415.pdf document : What regulation requires manufacturers to ensure vehicles minimize the risk of injury to occupants and other road users?

data_source: DocBench

task_type: mmreasoning

modality: vl


question: Here is a question from NYSE_TME_2021.pdf document : What percentage of revenue did social entertainment services contribute to the overall revenue in 2021?

data_source: DocBench

task_type: mmreasoning

modality: vl


question: Here is a question from 2020.acl-main.45.pdf document : By how much does the accuracy of BERT+CE on SST-2 exceed that of BERT+DL?

gold_answers: The accuracy of BERT+CE on SST-2 exceeds that of BERT+DL by 0.53.

data_source: DocBench

task_type: mmreasoning

modality: vl


question: Here is a question from NYSE_HSBC_2020.pdf document : What is the Group's net deferred tax asset in 2020?

data_source: DocBench

task_type: mmreasoning

modality: vl


question: Here is a question from W18-4401.pdf document : How many Facebook comments were provided in the dataset for training purposes?

data_source: DocBench

task_type: mmreasoning

modality: vl


question: Here is a question from NASDAQ_CMCSA_2021.pdf document : What segment contributed the most to the company's revenue in 2021?

gold_answers: Cable Communications contributed the most to the company's revenue in 2021, accounting for 53%.

data_source: DocBench

task_type: mmreasoning

modality: vl


question: Here is a question from 2020.sltu-1.28.pdf document : What is the total number of tokens in the Tamil-English language pair dataset?

gold_answers: The total number of tokens in the Tamil-English language pair dataset is 169,833.

data_source: DocBench

task_type: mmreasoning

modality: vl


question: Here is a question from 2023555905.pdf document : Are Home Office staff required to consider the maturity and actual age of an unaccompanied minor when assessing their asylum claim?

data_source: DocBench

task_type: mmreasoning

modality: vl


question: Here is a question from USCOURTS-laed-2_24-cv-00393-0.pdf document : Did the court grant the Plaintiff's emergency motion for a temporary restraining order?

data_source: DocBench

task_type: mmreasoning

modality: vl


question: Here is a question from P18-4013.pdf document : According to Figure 4, does the training speed of NCRF++ continue to increase after a batch size of 100?

gold_answers: Yes, the training speed of NCRF++ continues to increase even after a batch size of 100.

data_source: DocBench

task_type: mmreasoning

modality: vl


question: Here is a question from NYSE_ACN_2020.pdf document : What was the company's operating income for Europe in 2020?

gold_answers: The operating income for Europe in 2020 was $1,799 million.

data_source: DocBench

task_type: mmreasoning

modality: vl


question: Here is a question from NYSE_PEP_2020.pdf document : What was the company's net revenue for China in 2019?

gold_answers: The net revenue for China in 2019 was $1,300 million.

data_source: DocBench

task_type: mmreasoning

modality: vl


question: Here is a question from 2023555933.pdf document : Is there a fine for network operators in China who fail to authenticate users' identities?

data_source: DocBench

task_type: mmreasoning

modality: vl


question: Here is a question from USCOURTS-nmd-1_24-cr-00032-8.pdf document : What will happen if the government fails to establish the relevance of Exhibits 6F and 11C?

gold_answers: The Court will issue a curative instruction.

data_source: DocBench

task_type: mmreasoning

modality: vl


question: Here is a question from roche-holdings_2021.pdf document : Which region has the highest number of employees of the company and how many?

gold_answers: Europe has the highest number of employees with 43,181.

data_source: DocBench

task_type: mmreasoning

modality: vl


question: Here is a question from 2022666107.pdf document : When was the document publicly released?

data_source: DocBench

task_type: mmreasoning

modality: vl


question: Here is a question from NYSE_TM_2021.pdf document : What is the percentage of shares held by the largest shareholder listed in the report?

gold_answers: The largest shareholder listed is The Master Trust Bank of Japan, Ltd. with 11.71% of shares held.

data_source: DocBench

task_type: mmreasoning

modality: vl


question: Here is a question from 20220817.pdf document : How many Americans experience hearing loss, according to the studies cited by the FDA?

data_source: DocBench

task_type: mmreasoning

modality: vl


question: Here is a question from D18-1034.pdf document : What is the model performance on SentEval?

data_source: DocBench

task_type: mmreasoning

modality: vl


question: Here is a question from NYSE_CAT_2021.pdf document : What was the company's total sales figure for Energy & Transportation in the fourth quarter of 2021?

gold_answers: The total sales were $5.728 billion in the fourth quarter of 2021.

data_source: DocBench

task_type: mmreasoning

modality: vl


question: Here is a question from 20240106.pdf document : Has the FDA approved Florida's plan to import drugs from Canada?

data_source: DocBench

task_type: mmreasoning

modality: vl


question: Here is a question from FBS_F_Public-Accessible.pdf document : Is the Bureau aiming to establish and encourage evidence-based decision-making within senior leadership?

data_source: DocBench

task_type: mmreasoning

modality: vl


question: Here is a question from FBS_CSO_Public.pdf document : How does the Bureau plan to enhance its involvement in peace processes according to the strategic framework?

gold_answers: The Bureau plans to improve U.S. engagement in local and national peace processes by increasing CSO’s permanent capacity for practice-based technical expertise and advanced data analytics, as outlined in Bureau Objective 2.2.

data_source: DocBench

task_type: mmreasoning

modality: vl


question: Here is a question from NYSE_TME_2021.pdf document : What strategies is the company employing to maintain its competitiveness in the social entertainment market amid increasing regulatory oversight?

gold_answers: The company is working to maintain its competitiveness in the social entertainment market by engaging in ongoing product innovations and building additional verticals such as audio live streaming, expanding internationally, and offering new virtual interactive products.

data_source: DocBench

task_type: mmreasoning

modality: vl


question: Here is a question from NYSE_BRK-A_2021.pdf document : Did the revenues for the manufacturing, service, and retailing businesses of K-47 increase from 2020 to 2021?

data_source: DocBench

task_type: mmreasoning

modality: vl


question: Here is a question from OTC_NSRGY_2020.pdf document : Which market had the highest percentage decrease in sales in CHF during 2020 compared to 2019?

gold_answers: Brazil had the highest percentage decrease in sales in CHF during 2020 compared to 2019 with -23.5%.

data_source: DocBench

task_type: mmreasoning

modality: vl


question: Here is a question from NYSE_TM_2021.pdf document : What is the role of the Board of Directors in Toyota's Corporate Governance structure?

gold_answers: The Board of Directors is responsible for decision-making and management oversight within Toyota's Corporate Governance.

data_source: DocBench

task_type: mmreasoning

modality: vl


question: Here is a question from 20221031.pdf document : Is Dr. Frank Mitloehner associated with the University of California, Davis?

data_source: DocBench

task_type: mmreasoning

modality: vl


question: Here is a question from 2020.acl-main.423.pdf document : What is the model performance on SemEval-SS Frozen when SenseBERT is initalized from RoBERTa base?

gold_answers: The paper does not mention the performance of SenseBERT trained on RoBERTa.

data_source: DocBench

task_type: mmreasoning

modality: vl


question: Here is a question from 2020.emnlp-main.213.pdf document : What loss function is optimized in the Translation Ranking model architecture?

data_source: DocBench

task_type: mmreasoning

modality: vl


question: Here is a question from 2020.sltu-1.28.pdf document : What percentage of annotators had Tamil as the medium of schooling?

gold_answers: Approximately 45.45% of annotators had Tamil as the medium of schooling.

data_source: DocBench

task_type: mmreasoning

modality: vl


question: Here is a question from 2022666109.pdf document : What is the primary focus of Section V?

data_source: DocBench

task_type: mmreasoning

modality: vl


question: Here is a question from C18-1182.pdf document : According to Figure 1, how is each word in a sentence processed before being input to the LSTM layers?

gold_answers: Each word is processed through a word embedding layer which transforms the word into a numerical vector representation before being input to the LSTM layers.

data_source: DocBench

task_type: mmreasoning

modality: vl


question: Here is a question from P19-1041.pdf document : Which model's style space shows a clearer separation between different styles according to Figure 2?

gold_answers: The style space of DAE shows a clearer separation between different styles compared to VAE's style space.

data_source: DocBench

task_type: mmreasoning

modality: vl


question: Here is a question from 20220725.pdf document : Are conservative activists working to recruit county sheriffs to investigate elections?

data_source: DocBench

task_type: mmreasoning

modality: vl


question: Here is a question from W19-5302.pdf document : Is the Meteor++_2.0 (syntax+copy) score for the enkk-en language pair higher than the score for enfi-en?

data_source: DocBench

task_type: mmreasoning

modality: vl


question: Here is a question from N18-1012.pdf document : On which page does the paper present the Appendix?

data_source: DocBench

task_type: mmreasoning

modality: vl


question: Here is a question from NASDAQ_ATLX_2020.pdf document : How much was the annual revenue of Tesla in 2020?

data_source: DocBench

task_type: mmreasoning

modality: vl


question: Here is a question from FBS_GTM_18FEB2022_PUBLIC.pdf document : Is GTM collaborating with the Secretary’s Office of Diversity and Inclusion to implement the DEIA Strategic Plan?

data_source: DocBench

task_type: mmreasoning

modality: vl


question: Here is a question from ASX_LOV_2020.pdf document : Which country showed the largest increase in the number of Lovisa stores from 2018 to 2020?

gold_answers: The USA showed the largest increase in the number of Lovisa stores from 2018 to 2020, with an increase from 1 store to 48 stores.

data_source: DocBench

task_type: mmreasoning

modality: vl


question: Here is a question from 20231014.pdf document : What accomplishments had Louise Glück achieved before receiving the Nobel Prize?

gold_answers: She won the Pulitzer Prize and the National Book Award.

data_source: DocBench

task_type: mmreasoning

modality: vl