The US tech industry is endorsing a new plan this week called the ATOM Project, namely American Truly Open Models, which aims to win back the US lead in open-source artificial intelligence (AI) technology from China, the Washington Post reported on Wednesday. ... read full post

cross-posted from: https://lemmy.ml/post/34152716
cross-posted from: https://lemmy.ml/post/34037081
A note that this setup runs a 671B model in Q4 quantization at 3-4 TPS, running a Q8 would need something beefier. To run a 671B model in the original Q8 at 6-8 TPS you'd need a dual socket EPYC server motherboard with 768GB of RAM.