Here's the story not told in Nolan's Oppenheimer about those forced off their land in New Mexico

In 1942, the U.S. army gave 32 Hispano families on the Pajarito Plateau 48 hours to leave their homes and land, in some cases at gunpoint, to build the lab that would create the world's first atomic bombs, according to relatives of those removed and a former lab employee.

source https://www.cbc.ca/news/world/oppenheimer-new-mexico-land-removed-1.6922402?cmp=rss