transformers go brr but sometimes the loss curve looks like a roller coaster. we need more compute and ablation studies to really figure this thing out. reviewer 2 can fight me on this one.