kasrahabib/ReqSeek-bert-base-uncased

kasrahabib/bert-base-uncased-finetuned-iso29148-req-detector

This model is a fine-tuned version of google-bert/bert-base-uncased on an unknown dataset. It achieves the following results on the evaluation set:

• Train Loss: 0.0288
• Validation Loss: 0.4071
• Epoch: 29

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• optimizer: {'name': 'Adam', 'weight_decay': None, 'clipnorm': None, 'global_clipnorm': None, 'clipvalue': None, 'use_ema': False, 'ema_momentum': 0.99, 'ema_overwrite_frequency': None, 'jit_compile': True, 'is_legacy_optimizer': False, 'learning_rate': {'module': 'keras.optimizers.schedules', 'class_name': 'PolynomialDecay', 'config': {'initial_learning_rate': 2e-05, 'decay_steps': 3570, 'end_learning_rate': 0.0, 'power': 1.0, 'cycle': False, 'name': None}, 'registered_name': None}, 'beta_1': 0.9, 'beta_2': 0.999, 'epsilon': 1e-08, 'amsgrad': False}
• training_precision: float32

Training results

Train Loss	Validation Loss	Epoch
2.3106	1.6508	0
1.3653	1.0810	1
0.9467	0.8094	2
0.6660	0.6197	3
0.4734	0.5035	4
0.3338	0.4401	5
0.2284	0.4338	6
0.1593	0.4217	7
0.1243	0.4613	8
0.0996	0.4015	9
0.0889	0.4112	10
0.0772	0.4256	11
0.0662	0.3997	12
0.0626	0.4175	13
0.0566	0.3948	14
0.0569	0.3887	15
0.0508	0.4099	16
0.0472	0.3963	17
0.0485	0.3869	18
0.0437	0.4008	19
0.0452	0.4106	20
0.0406	0.4132	21
0.0391	0.3984	22
0.0353	0.4032	23
0.0331	0.4573	24
0.0336	0.4034	25
0.0353	0.4056	26
0.0309	0.4050	27
0.0303	0.4048	28
0.0288	0.4071	29

Framework versions

• Transformers 4.40.1
• TensorFlow 2.15.0
• Datasets 2.19.1
• Tokenizers 0.19.1