CHAPTER 05 / SOFTWARE AND SERVICE
What connects a model to the machines running it?
Computing software and services
Infrastructure software coordinates the work between an application and computing equipment. It includes operating systems, device drivers, libraries, scheduling, model serving and monitoring.
From request to computation
An application sends a request to serving software. Libraries express operations that processors can execute, while drivers connect software to particular devices. Operating systems manage resources and access. The exact stack varies with the model, hardware and deployment.
Sharing and operating resources
Schedulers assign work to available resources. Container orchestration systems such as Kubernetes can manage deployment and scaling, but are not required for every AI system. Cloud services package some of this equipment and software as rented capabilities; local deployments operate their own stack.
Reliability and visibility
Monitoring follows failures, resource use and response times. Access controls limit who can use a service or read its information. Updates, capacity planning and recovery procedures are continuing operational work. A working model alone is not a reliable service.
Keep this connection in mind.
The software stack turns available computation into an operable service. Its design depends on the workload.
- Deployed models served by Serving software
- Computing software runs on Computing hardware
Sources for this introduction
- Kubernetes · Overview ↗
Container orchestration, deployment and management of workloads; one example of infrastructure software.
- Google · Machine Learning Glossary ↗
Models, parameters, training, inference, datasets and evaluation.
Draft teaching text, prepared with AI assistance. Larry’s editorial review is pending. Read the editorial approach.
EXPLORE RELATED AREAS