Optimizing foundation models
Fine-Tuning
Fine-tuning tailors a general FM to a specific task or domain. It helps to:
- Increase specificity for domain nuances
- Improve accuracy on specialized tasks
- Reduce biases from the original training data
- Boost efficiency within a specific context
Fine-tuning approaches
| Approach | What it does | Good for |
|---|---|---|
| Instruction tuning | Retrains on prompts paired with desired outputs | Following commands: chatbots, virtual assistants |
| RLHF | Supervised training first, then reinforcement learning with a reward model built from human feedback | Aligning output with human values and preferences |
| Domain adaptation | Trains on an industry corpus, such as legal documents or medical records | Relevance and accuracy in one domain |
| Transfer learning | Reuses a model trained for one task as the starting point for another | Efficiency with less additional training |
| Continuous pretraining | Keeps feeding the model new data | Staying current with new vocabulary, trends, and research |
Metrics such as ROUGE and BLEU and techniques such as model pruning are not fine-tuning methods.
Pre-training data versus fine-tuning data
| Pre-training | Fine-tuning |
|---|---|
| Massive, diverse, broad coverage | Focused on the task |
| Built to generalize | Highly relevant to the desired outputs |
| Quantity | Quality over quantity: small, well-curated sets can go a long way |
Key steps to prepare fine-tuning data
- Data curation: rigorous selection of only relevant data
- Labeling: accurate, relevant labels are paramount because they drive specialization in the target domain
- Governance and compliance with industry regulations
- Representativeness and bias checking
- Feedback integration: user or expert feedback, especially for RLHF