| --- |
| license: mit |
| language: |
| - en |
| --- |
| ## Objective |
|
|
| Normalizer is a tool that processes various file types to extract prompt-response pairs for finetuning LLM models on unstructured data. |
|
|
| The end-state will be a GUI where users of any technical level can drag and drop multiple files at once into the application, then they hit a 'create fine-tuning data' button and they'll receive a .csv with a system prompt and a series of matched pair prompt-completion responses. |
|
|
| ## Working Repository |
|
|
| https://github.com/anarchy-ai/normalizer |
|
|
| ## Input Formats |
|
|
| - **Text**: .txt, .md, .mdx |
| - **Documents**: .pdf, .doc, .docx |
| - **Spreadsheets**: .xlsx, .csv |
| - **Code**: .py, .js, .html, .java |
| - **Images**: .jpg, .png (will require OCR) |
|
|
| ## Output Format |
|
|
| A .csv file with 3 columns: |
|
|
| | System Prompt | User Prompt | Response | |
| | -------------------------------------- | ------------------------------------- | ---------------------------------------------| |
| | Hi, how can I help you today? | | |
| | | What do these lab results suggest? | These lab results suggest that the patient is healthy, as no anomalous data has been detected. | |
| | | What is the sentiment of the last 5 customers who came into support chat? | The last five customers have a neutral to positive sentiment. | |
| | | ... | ... | |
|
|
| ## Required Libraries |
|
|
| ```pip install PyPDF2 python-docx pandas openpyxl pillow pytesseract beautifulsoup4 transformers datasets``` |
|
|
| ## Project Structure |
|
|
| ```CSS |
| normalizer/ |
| │ |
| ├── src/ |
| │ ├── __init__.py |
| │ ├── file_ingest.py |
| │ ├── prompt_extractor.py |
| │ ├── main.py |
| │ |
| ├── app.py |
| ├── requirements.txt |
| ├── README.md |
| └── .gitignore |
| ``` |
