Skip to main content
Search
Join
Log in
Mentorship
Join CCMNet
Mentorship Opportunities
Community
People
CCMNet Members
Affinity Groups
Blog
Jobs
Organizations
Coco, Community of Communities
Join the CSSN
Get Help
Ask a Question
CI Links
Request a Consult
Projects
Knowledge Base
Ask.CI Forum
KB Resources
Tags
About Us
User Guide
Become a Campus Champion
User Guide
Affinity Groups FAQ
Governance
Code of Conduct
News
CCMNet Guide
About CCMNet
Tags
A green, open-source greater than 10B parameter language model.
Project Information
Project Status:
Halted
Project Region:
CAREERS
Submitted By:
Chris Hill
Project Email:
Cnh@mit.edu
Project Institution:
MIT
Anchor Institution:
NE-MGHPCC
Project Address:
Cambridge, Massachusetts. 02139
Mentors:
Chris Hill
Project Description
We are developing a new language model derived from the EleutherAI GPT-Neo initiative ( https://github.com/EleutherAI/gpt-neo ) for application to two projects. These problems need models the skill close to that of the state-of-the art GPT-3 proprietary model. One project is a demonstration of the model for state-of-the-art image captioning, the other project is the publication of the full model as an open community tools for the research community.
For both projects we are interested in collaborating with Cyberteams students to work on model training optimization and testing. The project is looking to run model training and evaluate performance on multi-node configurations of the Aimos 6-GPU/node system. This will allow us to examine scaling and potentially prepare for large experiments with appropriate discussions with IBM teams. The model we will use is efficient and some preliminary work has been undertaken at MGHPCC. Both the RPI and MGHPCC systems have excellent carbon emissions footprints so we also anticipate being able to report energy and emissions statistics that are state-of-the-art for large scale language model training too.
Project Information
Project Status:
Halted
Project Region:
CAREERS
Submitted By:
Chris Hill
Project Email:
Cnh@mit.edu
Project Institution:
MIT
Anchor Institution:
NE-MGHPCC
Project Address:
Cambridge, Massachusetts. 02139
Mentors:
Chris Hill
Project Description
We are developing a new language model derived from the EleutherAI GPT-Neo initiative ( https://github.com/EleutherAI/gpt-neo ) for application to two projects. These problems need models the skill close to that of the state-of-the art GPT-3 proprietary model. One project is a demonstration of the model for state-of-the-art image captioning, the other project is the publication of the full model as an open community tools for the research community.
For both projects we are interested in collaborating with Cyberteams students to work on model training optimization and testing. The project is looking to run model training and evaluate performance on multi-node configurations of the Aimos 6-GPU/node system. This will allow us to examine scaling and potentially prepare for large experiments with appropriate discussions with IBM teams. The model we will use is efficient and some preliminary work has been undertaken at MGHPCC. Both the RPI and MGHPCC systems have excellent carbon emissions footprints so we also anticipate being able to report energy and emissions statistics that are state-of-the-art for large scale language model training too.