Updating documentation

README.md

@@ -14,37 +14,52 @@ pinned: false
 
 # Metric Card for regression_evaluate
 
-***Module Card Instructions:*** *Fill out the following subsections. Feel free to take a look at existing metric cards if you'd like examples.*
-
 ## Metric Description
-*Give a brief overview of this metric, including what task(s) it is usually used for, if any.*
+
+This metric aims to evaluate regression tasks done by LMs. It expects the model to generate a list of numerical values to compare it to gold list of numerical values.
 
 ## How to Use
-*Give general statement of how to use the metric*
 
-*Provide simplest possible example for using the metric*
+This metric takes 2 mandatory arguments : `generations` (a list of string), `golds` (a list of list of floats).
+
+```python
+import evaluate
+metric = evaluate.load("rfr2003/regression_evaluate")
+results = metric.compute(generations=['[150, 0]'], golds=[183, 177, 146, 85, 70, 78, 55, 17, 0, -1, -1])
+print(results)
+{'precision': [4.0], 'recall': [344.0], 'macro-mean': [174.0], 'median macro-mean': 174.0}
+```
+
+This metric accepts one optional argument:
 
-### Inputs
-*List all input arguments in the format below*
-- **input_field** *(type): Definition of input, with explanation if necessary. State any default value(s).*
+`d`: function used to compute the distance between a generated value and a gold one. The default value is a function computing the absolute difference between two numbers.
 
 ### Output Values
 
-*Explain what this metric outputs and provide an example of what the metric output looks like. Modules should return a dictionary with one or multiple key-value pairs, e.g. {"bleu" : 6.02}*
+This metric outputs a dictionary with the following values:
 
-*State the range of possible values that the metric's output can take, as well as what in that range is considered good. For example: "This metric can take on any value between 0 and 100, inclusive. Higher scores are better."*
+`precision`: Sum of the minimum distances between each predicted value and the set of gold values, computed for each question.
+
+`recall`: Sum of the minimum distances between each gold value and the set of generated values, computed for each question.
+
+`macro-mean`: Average between precision and recall, computed for each question.
+
+`median macro-mean`: Median accross macro-mean values.
 
 #### Values from Popular Papers
-*Give examples, preferrably with links to leaderboards or publications, to papers that have reported this metric, along with the values they have reported.*
 
 ### Examples
-*Give code examples of the metric being used. Try to include examples that clear up any potential ambiguity left from the metric description above. If possible, provide a range of examples that show both typical and atypical results, as well as examples where a variety of input parameters are passed.*
+
+```python
+import evaluate
+metric = evaluate.load("rfr2003/regression_evaluate")
+results = metric.compute(generations=['[150, 0]'], golds=[183, 177, 146, 85, 70, 78, 55, 17, 0, -1, -1])
+print(results)
+{'precision': [4.0], 'recall': [344.0], 'macro-mean': [174.0], 'median macro-mean': 174.0}
+```
 
 ## Limitations and Bias
-*Note any known limitations or biases that the metric has, with links and references if possible.*
 
 ## Citation
-*Cite the source where this metric was introduced.*
 
-## Further References
-*Add any useful further references.*
+## Further References

regression_evaluate.py

@@ -37,22 +37,25 @@ This metric aims to evaluate regression tasks done by LMs.
 
 # TODO: Add description of the arguments of the module here
 _KWARGS_DESCRIPTION = """
-Calculates Accuracy and Blue-1 between generations and gold answers in a MCQ context.
+Calculates Precision, recall and macro-mean between generations and gold answers in a regression context.
 Args:
     generations: list of predictions to score. Each predictions
         should be a string generated by a LM model.
     golds: list of reference for each prediction. Each
         reference should be a list of floats.
+    d: function used to compute the distance between a generated value and a gold one.
 Returns:
-    precision:  ,
-    recall:
+    precision: Sum of the minimum distances between each predicted value and the set of gold values, computed for each question.
+    recall: Sum of the minimum distances between each gold value and the set of generated values, computed for each question.
+    macro-mean: Average between precision and recall, computed for each question.
+    median macro-mean: Median accross macro-mean values.
 Examples:
     Here is an exemple on how to use the metric:
 
     >>> metric = evaluate.load("rfr2003/regression_evaluate")
     >>> results = metric.compute(generations=['[150, 0]'], golds=[183, 177, 146, 85, 70, 78, 55, 17, 0, -1, -1])
     >>> print(results)
-    {'precision': 4.0, 'recall': 344.0, 'macro-mean': 174.0, 'median macro-mean': 174.0}
+    {'precision': [4.0], 'recall': [344.0], 'macro-mean': [174.0], 'median macro-mean': 174.0}
 """
 
 
@@ -91,13 +94,13 @@ class regression_evaluate(evaluate.Metric):
             dists.append(g_dist)
     
         dists = np.array(dists)
-        precision = np.min(dists, axis=0).sum()
+        precision = np.min(dists, axis=0).sum().item()
     
-        recall = np.min(dists, axis=1).sum()
+        recall = np.min(dists, axis=1).sum().item()
     
         return precision, recall
 
-    def _compute(self, generations, golds):
+    def _compute(self, generations, golds, d=lambda x,y: abs(x-y)):
         assert len(generations) == len(golds)
         assert isinstance(golds, list)
     
@@ -110,21 +113,18 @@ class regression_evaluate(evaluate.Metric):
                 
             f_ans = list(set([float(a.replace(',', '')) for a in f_ans])) #get rid of duples values
     
-            precision, recall = self._calculate_pre_rec(f_ans, f_gold, lambda x,y: abs(x-y))
+            precision, recall = self._calculate_pre_rec(f_ans, f_gold, d)
     
             precisions.append(precision)
             recalls.append(recall)
             means_pre_rec.append((precision+recall)/2)
             
     
-        macro_prec = np.mean(precisions).item()
-        macro_rec = np.mean(recalls).item()
-    
         metrics = {}
         metrics.update({ 
-            'precision': np.mean(precisions).item(),
-            'recall': np.mean(recalls).item(),
-            'macro-mean': np.mean(means_pre_rec).item(),
+            'precision': precisions,
+            'recall': recalls,
+            'macro-mean': means_pre_rec,
             'median macro-mean': median(means_pre_rec)
         })
     

tests.py

@@ -2,6 +2,6 @@ test_cases = [
     {
         'generations': ['[150, 0]'],
         'golds': [183, 177, 146, 85, 70, 78, 55, 17, 0, -1, -1],
-        "result": {'precision': 4.0, 'recall': 344.0, 'macro-mean': 174.0, 'median macro-mean': 174.0}
+        "result": {'precision': [4.0], 'recall': [344.0], 'macro-mean': [174.0], 'median macro-mean': 174.0}
     }
 ]