Compute/memory requirements scripts

Estimations for memory and compute requirements for training large language models.

scripts/memory_compute_estimations/compute_req.py

 setups = {
+    "35M": { "L": 6, "H": 8, "D": 512, },
     "70M": { "L": 10, "H": 10, "D": 640, },
     "284M": { "L": 20, "H": 16, "D": 1024, },
     "512M": { "L": 24, "H": 10, "D": 1280, },