ECE 364

https://courses.grainger.illinois.edu/ece364/fa2025

This course covers how to use auto-differentiation tools like PyTorch for basic linear algebra operations, how to leverage them for basic machine learning algorithms (linear regression, logistic regression, deep nets, k-means), and how to extend them with custom methods. Auto-differentiation tools are key for data analysis and a solid understanding is increasingly important across disciplines. Different from courses which focus on algorithmic and theoretical aspects of Machine Learning, here we focus on deployment of auto-diff tools to different areas of interest. Core concepts in machine learning are introduced and developed both mathematically and practically in software within the framework of implementation in a popular auto-differentiation framework like PyTorch.

Outline of covered topics:

1. Introduction (week 1-2): introduction to software, tensors, tensor views and functions
2. Linear algebra, differentiation, and optimization (week 3-5): introduction to basic linear algebra, calculus, differentiation, and differentiation w.r.t. vectors, as well as their implementation in auto-diff tools and their use in basic optimization algorithms
3. Linear Regression (week 6-8): careful discussion of linear regression, its implementation in auto-diff tools and its combination with auto-diff tool primitives like datasets, data-loaders etc.
4. Logistic Regression, Deep Nets and Temporal Data (week 9-11): mathematical derivation of logistic regression, discussion of its combination with auto-diff tool primitives and extension to deep nets, image data and temporal data
5. Clustering (week 12): discussion of basic unsupervised learning algorithms like k-means clustering and its implementation with PyTorch primitives
6. Extending PyTorch and Special Topics (week 13-14): discussion of distributed training, development of extensions via Python, C++ and CUDA, or other special topics based on students and instructor interest.
7. Review lectures (one week total): we need one week to review material before the midterm and the final

Detailed Outline:

Students are completing assignments on their laptop and/or the google cloud platform.

The course may have an optional or required final project where students employ their knowledge from the semester on a dataset and machine learning task of their choice.

Linear algebra (MATH 257)

No text book is used.

This course is a technical elective for both electrical engineers and computer engineers. The course counts as a department-approved software lab course.

This course covers how to use auto-differentiation tools like PyTorch for basic linear algebra operations, how to leverage them for basic machine learning algorithms (linear regression, logistic regression, deep nets, k-means), and how to extend them with custom methods. Auto-differentiation tools are key for data analysis and a solid understanding is increasingly important across disciplines. Different from courses which focus on algorithmic and theoretical aspects of Machine Learning, here we focus on deployment of auto-diff tools to different areas of interest. Core concepts in machine learning are introduced and developed both mathematically and practically in software within the framework of implementation in a popular auto-differentiation framework like PyTorch.

After completing this course, students should:

• Be familiar with key primitive PyTorch data types, e.g. tensors, datasets, dataloaders, modules, and be able to design and implement common or custom instances of each type for a given data source, machine learning model, or optimization algorithm. (1, 2, 6, 7)
• Be able to explain the mathematical motivation and process of backpropagation for accumulating gradients in computational graphs (6, 7)
• Be able to apply auto-differentiation tools effectively using PyTorch to update trainable parameters in a computational graph (1, 2)
• Understand how auto-differentation tools, computational graphs, and machine learning algorithms can be used together to efficiently solve data-driven engineering problems (1, 2, 6, 7)
• Understand the application settings for linear regression models, mathematical principles of their solutions, and efficient implementations of their solutions via auto-differentiation in PyTorch. (1, 2, 6, 7)
• Understand the application settings for logistic regression models, mathematical principles of their solutions, efficient implementations of their solutions via auto-differentiation in PyTorch, and the key differences between logistic and linear regression. (1, 2, 6, 7)
• Be able to explain how neural networks, e.g. multi-layer perceptrons, convolutional neural networks, recurrent neural networks, extend linear and logistic regression models to more complex machine learning tasks and necessary non-linear solutions. (1, 6, 7)
• Understand how objective functions may be chosen for desired engineering problems alongside appropriate machine learning algorithms (1, 2, 6, 7)
• Be able to identify the key differences and use cases of common neural network architectures and models for engineering and data science problems (1, 6)
• Be able to design and implement existing or custom solutions for popular neural network architectures, e.g. convolutional neural networks and transformers, using PyTorch and autodifferentation for engineering and data science problems (1, 2, 6)
• Be comfortable reading PyTorch and other software package documentation as necessary to apply alternative solutions or for experimentation within class homeworks or projects. (7)
• Understand how PyTorch may be extended to further implementations in C, C++, other Python libraries, and/or CUDA. (1, 2, 7)