Back to LING 385
Lecture 1
Goals of LING 385
- learn the ideas behind the computations that have allowed modern language technologies to become important parts of society
- and the practical computations that implement the ideas
- learn how scientific ideas from different centuries are creatively put together to make new seemingly revolutionary ideas
- learn about how commonly used tools actually work
Language, Technology, and Society
- great deal of what we hear about language technologies is the harm they can cause
- very valid concerns
- however, technologies can also be used for good
- understanding how these systems work and acting on this understanding is the best way to make sure they are used for good causes
Language Technology: how much do we know?
- most current systems are open-source
- we know every mathematical idea and algorithm powering them
- despite this, we don't really know how it all comes together on the lowest level
Neural Networks: an Interdisciplinary Discipline
- interest in NNs waned till the physicists got interested in late 70’s
- great deal of physics is about figuring out the macroscopic properties of systems (e.g., gases, polymers, magnets) from the microscopic laws governing atoms and molecules
- Hopfield and Smolensky saw the analogy of molecules to neurons and macroscopic material properties to cognitive processes
- in 1986, when Rumelhart and McClelland published Parallel Distributed Processing
- brought together work by psychologists, computer scientists, and physicists
- after that, NN research waned again
- in 2006, deep learning (using neural networks in industry) was born
- now, people have started trying to understand how these systems work by going back to Hopfield, Rumelhart and McClelland
Associative Memory
- based on associations of two things (or between part or whole)
- John Hopfield, in 1982, showed that some very simple calculations, analogous to ones in physics, can model associative memory
- 40 years later, his theory was re-used, 2021–current, to try to figure out how systems like ChatGPT actually work
- heteroassociative memory = remembering some piece of information from another
- DALL-E is heteroassociative (text → image)
- auto-associative memory = remembering a whole from a part, or a noisy version of the whole
- ChatGPT is auto-associative (text → text)