Move to HuggingFace datasets
This should be much easier to work with, as we don't have to make a separate dataset each time. HuggingFace datasets also has nice functionality which we can use, without loss of performance.
parent
cb1a3343
No related branches found
No related tags found
Showing
- optimus/dataloader.py 1 addition, 1 deletionoptimus/dataloader.py
- optimus/datasets/__init__.py 0 additions, 2 deletionsoptimus/datasets/__init__.py
- optimus/datasets/dataset_utils.py 0 additions, 62 deletionsoptimus/datasets/dataset_utils.py
- optimus/datasets/tinystories.py 0 additions, 130 deletionsoptimus/datasets/tinystories.py
- optimus/datasets/wikitext103.py 0 additions, 120 deletionsoptimus/datasets/wikitext103.py
- training.py 26 additions, 4 deletionstraining.py
Loading
Please register or sign in to comment