← All self-study courses

See how AI learns — hands on (10 sessions, 15 minutes each, v1.2)

Goal of this course: Watch AI learn with short programs, then read a sleep-scoring AI.

This course is for university and high school students who have learned high school mathematics. By reading, running, and changing exactly one part of short Python programs yourself, you will see how an AI "reads data," "learns," and "makes decisions." The last three sessions use a sleep-scoring AI that this laboratory has published as a paper and whose program is publicly available. After the ten sessions, you will be able to read an AI built by someone else from its paper and documentation, and to try for yourself what happens when you change a particular part.

A Japanese version of this course (講座_AIの動作原理_v1.2.md) is also available; to study in Japanese, use that file or tell the AI "Please teach this course in Japanese."

Introduction (How to Use This Course)

What this is

This is a set of teaching materials published by the Division of Neurophysiology, Kobe University Graduate School of Medicine, for students' self-study. In 15 minutes a day over 10 sessions, you will learn how AI (artificial intelligence) works by reading, running, and modifying programs. It is not a textbook you simply read; it is a course in which a conversational AI acts as your teacher and you progress by answering its questions.

The route through this course is based on the path that a faculty member of this laboratory actually followed in learning to read and modify machine learning programs (machine learning: methods that make a computer derive rules from data) on their own. The goal is not "being able to build everything from scratch" but "being able to read an AI built by someone else, change one part, and check the effect." In Sessions 8-10, you will read an AI that this laboratory published in 2021 and whose program is publicly available: it determines the sleep stage from the electroencephalogram (EEG) of mice (the lab's nickname for it is "the Tezuka AI"). You will then change one part of a small model built on the same idea and measure the effect.

Who this is for (assumed background)

• High school mathematics (linear functions, means, ratios) is enough.
• You can proceed without knowing calculus. Session 6 uses the idea of "nudging a value slightly and seeing whether the error goes up or down."
• It is fine if you have never programmed before. In Session 1 you will learn only the minimum syntax you need.

What you need

• A conversational AI (one that answers your questions in a chat). One that can read uploaded files is convenient.
• Paper, pencil, and a calculator (a smartphone calculator is enough).
• An environment in which you can run Python (used from Session 1). A free environment that runs in your browser is fine. For example, on the "Try Jupyter" page of the official Jupyter website (https://jupyter.org/try ), choose "JupyterLab" and Python will run inside your browser without installing anything.
• The tools (libraries = collections of handy functions) we use are numpy (numerical computation), pandas (table computation), matplotlib (drawing figures), and scikit-learn (a machine learning toolbox; Sessions 7, 9, and 10). Apart from the public dataset in Session 7, all data are generated by the programs in this course.
• On days when you cannot run Python, you can still make progress by asking the AI to "Please show me an example of the output" and thinking about it.

How to proceed

1. Load this file into a conversational AI (attach the file, or paste the full text).
2. Type "Please start Session 1."
3. Answer the AI's questions in your own words. It is fine to make mistakes. The places where you make mistakes are the places where you learn the most.
4. Stop after 15 minutes. The next day, load this file again and type "Please start Session 2."

You may do one session a day for 10 days, or several at once, but spacing the sessions out helps you remember them better.

Cautions

• AI can make mistakes. Check important points against textbooks or original sources. Session 10 also deals with the importance of checking numbers against the original source.
• Do not enter your own or anyone else's personal information, medical information, or other people's data into the AI. This course is designed so that you can complete it using only synthetic data and public data.
• This course is not intended to provide medical advice.

Request for feedback (please read)

We revise this course every time based on the comments of the people who use it. If you have used it, please be sure to send us feedback.

Send to: the address listed on the "Contact" page of the laboratory website Subject: "Course feedback: How AI Works" What we would like you to write (4 points): 1. How many sessions you completed 2. What was hard to understand (session and location) 3. What was good 4. How long one session took (approximately)

Comments from people who stopped partway are also very helpful.

Terms of use

• You are free to use this course for personal study and in classes.
• If you distribute a modified version, please contact the laboratory.
• The content is current as of November 2026.

Instructions for the AI (How to Act as the Teacher)

Everything below is addressed to the AI that has loaded this file. Learners are welcome to read it as well.

You (the AI) are the teacher for this course. Please proceed as follows.

1. Keep each session to 15 minutes. As a guide: 5 minutes of explanation, 7 minutes of hands-on exercises, and 3 minutes of review. Cover only the session the learner asked for with "Please start Session N."
2. At the start of each session, state where the learner is in one line. Example: "Session 4 of 10. So far, you have learned to read a table, remove missing values, and make a figure."
3. Ask only one question at a time and wait for the answer. Do not ask two or more questions at once.
4. Do not give the answer first. Use the "Answers and reasoning (for the AI)" section of each session only after the learner has answered on their own, or after they say they still do not know even after thinking it over twice, and use only the parts needed. Do not paste it as is.
5. Do not ask questions that cannot be solved without syntax that has not yet appeared. Questions must be solvable using only what is listed under "New syntax in this session" and what appeared in earlier sessions. When you use a new function, add a brief note "name = what it does" before using it. Things that appeared in earlier sessions may be used without explanation.
6. Hints should be parts only. If the learner gets stuck, point to the name of the function to use or the line to look at. Do not write the full answer code. The learner does the assembling.
7. If the learner seems lost, rephrase one level more simply. Do so in one of these ways: replace it with a familiar example, make the numbers smaller, or describe a figure in words.
8. At the end of each session, give three key points and two short review questions. The review questions may be taken from the session's "Review questions."
9. Do not move on to the next session on your own. When a session is over, say "That's all for today. Next time, type 'Please start Session N+1.'" and stop.
10. If the learner writes about a medical concern or personal information, politely decline to handle it. Say "This course cannot deal with medical consultations or personal information. It is safer to delete what you entered," and return to the course. Do not ask for the learner's name, school, or contact details.
11. At the start of the first session, ask only two things. (1) How confident they are in high school mathematics (linear functions, means). (2) Whether they have ever written a program. Adjust the level of detail of your explanations to the answers.
12. Learners run the code themselves. For those who cannot run it, show the session's "Example output" and have them think about what the numbers mean. The output numbers may differ slightly depending on the environment. When an error occurs, first read the last line of the error together. If the error says a name does not exist, as in name 'pd' is not defined, it is usually because the Python session in the browser has lost the variables you defined earlier (time has passed, or the page was reopened), so tell the learner to rerun the cells in order from the top.
13. If asked about facts not in the text, answer only within what you are sure of. If you are not sure, say "I am not certain," and show how to check (textbooks, original sources, databases).
14. At the end of each session, add a single line asking for feedback ("Please send us your impressions via the Contact page of the laboratory website"). In Session 10, carefully guide the learner through the four points in "Request for feedback" in the Introduction.

Session 1: First Steps in Python (about 15 minutes)

Goals of this session

1. Run Python in a browser and execute a cell (a box in which you write code).
2. Explain in your own words five things: variables, lists, for, if, and functions.
3. Write on your own a program that "displays only the values greater than the mean."

Explanation

Do you need to memorize an entire programming language to be able to read AI programs? The answer is no. Almost all of the syntax used in the ten sessions of this course is a combination of today's five things.

In Sessions 1-6, we use a single synthetic dataset throughout: numbers recorded every second by a classroom sound level meter (a device that measures loudness). The unit is dB (decibels, a unit of loudness), and we assume that a quiet class is around 45 dB and a noisy break is around 65 dB. The idea you learn here, "cut out the last few seconds and make a decision," is used as is in the laboratory's sleep-scoring AI that you will read in Sessions 8-10.

Today's five pieces of syntax are as follows.

• Variable: a box that gives a value a name so it can be remembered. If you write mean = 52.9, you can afterwards retrieve 52.9 with mean.
• List: values lined up in order. They are enclosed in square brackets, as in [44, 46, 45].
• for: takes the items of a list out one at a time and repeats the same thing. The colon : at the end of the line and the indentation (four spaces) of the next line mark "repeat from here to here."
• if: executes only when a condition holds. This also uses a colon and indentation.
• Function: a procedure bundled together and given a name. You create it with (a line of program) and send back the answer with (a line of program).

Parentheses () are the signal "execute this now." len(levels) means "count the items in levels, right now."

Python in the browser loses the variables it has defined after some time has passed or when the page is reopened. If you get an error saying a name does not exist, rerun the cells in order from the top.

New syntax in this session: listed in the file for the AI (.md).

Hands-on exercise

In JupyterLab, create a new "Python" notebook, paste the following code into one cell, and run it (press Enter while holding down the Shift key).

Exercise to write yourself: Using the same levels, write in a new cell a program that displays only the values greater than the mean (mean). Use for, if, and print.

Review questions

1. What do you think happens if you delete the colon at the end of the line (a line of program)?
2. If you change the 55 in the function is_loud to 64, how many lines are displayed?
3. What is the difference between (a line of program) and a line containing only mean? (Hint: only the last line of a cell is displayed even without print.)

Summary

• A variable is a named box, a list is a sequence, for is repetition, if is a condition, and a function is a bundle of steps.
• The colon : and indentation express "from where to where." Parentheses () are the signal "execute now."
• Python in the browser may lose what it has defined. When in trouble, rerun the cells in order from the top.

Session 2: Computing All at Once and Cutting Out a Part (numpy) (about 15 minutes)

Goals of this session

1. Use numpy arrays (sequences of numbers) to compute on all values at once without writing a for loop.
2. Read the shape of an array.
3. Cut out a part with a slice [start:end], and explain that this becomes a "window."

Explanation

To subtract 40 from each of 10 values, do you need to repeat 10 times with a for loop? With numpy, one line is enough.

numpy is a tool that handles sequences of numbers as "arrays." If you write - 40 after an array, 40 is subtracted from all values at once. Computing on a whole sequence at once like this, without repeating item by item, is called vectorization. Because it is fast and easy to read, almost all AI programs are written this way.

The shape of an array is displayed like (10,). This means "10 values lined up in a single line." For an image, two or more numbers appear, as in (8, 8) (8 high × 8 wide).

Another important thing is the slice (cutting out). levels[0:5] cuts out the five items "from item 0 up to, but not including, item 5." Numbering starts at 0, and the end number is not included.

This operation of "cutting out the first 5 seconds" is in fact the central idea of Sessions 8-10. When an AI makes decisions on data that arrive over time, it usually cuts out "the most recent few seconds" and decides on that. This cut-out range is called a window. Today's slice is exactly that window.

New syntax in this session: listed in the file for the AI (.md).

Hands-on exercise

Exercise to write yourself: Cut out and display the second window (the five items from second 5 to second 9), and display its mean as well. Which window is closer to "noisy"?

Review questions

1. What do you think happens if you compute - 40 on the Session 1 list [44, 46, ...] as it is?
2. Which values come out with levels[2:4]?
3. levels.shape has no (), but levels.mean() does. What is the difference?

Summary

• numpy arrays let you compute on everything at once without writing a for loop (vectorization).
• shape is the form of an array. A slice [start:end] counts from 0 and does not include end.
• A cut-out of "the most recent few seconds" is a window. It is the basic building block when an AI makes decisions on time-series data.

Session 3: Handling Data as a Table and Making Figures (pandas and matplotlib) (about 15 minutes)

Goals of this session

1. Read a CSV (a table file separated by commas) with pandas and check its shape and contents.
2. Find and remove missing values, and compute means by group.
3. Draw a figure with matplotlib, specifying the horizontal and vertical axes.

Explanation

How often does research data arrive as a "clean table" from the start? Hardly ever. The measuring device may malfunction and values may drop out, or recording may stop partway. The first job, before passing data to an AI, is to read it as a table and check where values are missing.

pandas is a tool that adds "column names" to numpy arrays so they can be handled as tables. Such a table is called a DataFrame, and it is often stored in a variable named df.

This time, we first create a CSV file with a function provided in this course and then read it in. It is a 60-second noise recording with three columns: sec (which second), level (loudness), and state (class or break).

Missing values are represented by NaN (Not a Number = a mark meaning there is no value). df.isna().sum() counts the missing values in each column, and df.dropna() removes the rows containing missing values. Then groupby (grouping) gives "the mean during class and the mean during the break." Many tables of research results have this form.

Finally, we draw a figure. Things you might overlook in a table of numbers become obvious at a glance in a figure.

New syntax in this session: listed in the file for the AI (.md).

Hands-on exercise

Run the following three cells in order from the top.

(The contents of make_noise_csv exist only to create the data. It is fine if you cannot read them yet.)

(This cell displays no text; one figure appears. The line should be high only between seconds 20 and 39. The text in the figure is in English because, in some environments, Japanese characters can turn into empty boxes.)

Exercise to write yourself: Change the last line of the second cell slightly to display the maximum for class and for the break (hint: the mean is mean, the maximum is max).

Review questions

1. Even after running df.dropna(), df still has 60 rows. Why?
2. If, instead of just one missing value, 10 consecutive seconds were missing in the middle of the break, how could the groupby means change?
3. If you write (a line of program) without passing df2["sec"] as the horizontal axis, what do you think the horizontal axis becomes?

Summary

• pandas works with tables that have named columns. After reading one, first check its shape (shape) and missing values (isna).
• dropna returns a "new table" without the rows with missing values. groupby lets you summarize by group.
• Figures are drawn by specifying x and y. What you overlook in a table of numbers is obvious at a glance in a figure.

Session 4: Before Smoothing, Decide the Unit of the Window (Preprocessing) (about 15 minutes)

Goals of this session

1. Smooth jagged values with a moving average (averaging while sliding a window).
2. Explain why the result is NaN until the window is filled.
3. Show with numbers that, when values are missing, a "5-row window" and a "5-second window" give different results.

Explanation

Preparing data before passing it to an AI is called preprocessing. Preprocessing may look unglamorous, but it is the part that determines how well the AI performs. There are three main reasons: values are missing; noise from the measuring device makes the data jagged; and the scales of the values are not aligned. This time we deal with the second one, "smoothing."

The standard method of smoothing is the moving average. You take the mean of five consecutive values, shift by one, take the mean of five again, and repeat. This is the same as sliding the "window" from Session 2 one step at a time. In pandas you write .rolling(window=5).mean(). For the first four rows, five earlier values are not yet available, so the result is NaN (no value).

Here is a question. If the measuring device malfunctioned and the records from second 8 to second 13 were missing as whole rows, is it still fine to average with a "5-row window"?

The answer is no. Because the rows close up, "5 rows" no longer means "5 seconds." The quiet value at second 7 and the noisy value at second 14 end up in the same window. You skip over what happened in between (the bell rang) and produce a mean that mixes quiet and noisy.

In such cases, you define the window not by "the number of rows" but by "the length of time." In pandas, writing df.rolling("5s", on="t") averages only "the values that actually exist within the 5 seconds up to that point."

There are two important rules in preprocessing: the order of "removing" and "applying a window", and the unit of the window (number of rows or time). Each of them produces different numbers from the same data.

New syntax in this session: listed in the file for the AI (.md).

Hands-on exercise

The 49.0 in row 4 is the same as "the mean of the first 5-second window" that you calculated in Session 2.

Next is data in which seconds 8-13 are missing as whole rows.

Exercise to think through yourself: Look at the row for second 14. Of which values are the 49.2 of the 5-row window (row5) and the 65.0 of the 5-second window (time5) each the mean? Check by calculating by hand.

(If you have time: writing plt.plot from Session 3 twice in a row lets you draw two lines overlaid in one figure. Overlay level and smooth from the first exercise and see what the smoothed line looks like.)

Review questions

1. If you change window=5 to window=3 in the first code, for how many initial rows is the result NaN?
2. In the first four rows, row5 is NaN, yet time5 has values. Why?
3. What is dangerous about "removing the missing values first and then applying a window defined by a number of rows"? Answer in one sentence.

Summary

• A moving average takes the mean while sliding the window one step at a time. Until the window is filled, the result is NaN.
• When values are missing, "N rows" is no longer "N seconds." A window defined by time averages only the values that actually exist.
• In preprocessing, the "order" and the "unit of the window" change the result. Always be able to say what you averaged.

Session 5: How Do We Measure an AI's Performance? (about 15 minutes)

Goals of this session

1. Explain, using exam preparation as an analogy, why data are split into "training, validation, and test" sets.
2. Calculate the positive predictive value and sensitivity from a confusion matrix (the four cells of hits and errors).
3. Explain, with an example, situations in which "looking only at accuracy is dangerous."

Explanation

In this session you will not write any programs. It is the session about the "yardsticks" for measuring how good an AI is. We proceed by hand calculation.

1. Training, validation, and test When building an AI, we split the data into three parts.

• Training (train): the data the AI learns from. This corresponds to studying with a textbook.
• Validation: the data used to choose settings (window length, decision threshold, and so on). This corresponds to planning your strategy with mock exams.
• Test: the data used only once at the end to measure performance. This is the real exam.

If you measure performance on the data used for learning, even rote memorization gets full marks. That does not answer the question "Can it be used on data it has never seen?"

Here there is a rule that is easy to overlook. The split is made not by "rows" but by units such as "people" or "places." For example, suppose you recorded noise in five classrooms. Recordings from the same classroom carry habits peculiar to that classroom, such as the sound of the air conditioner. If you scatter the one-second rows randomly into training and test, rows from the same classroom also end up in the test set, and the AI gets the answers right by memorizing the classroom's habits. This is called leakage (the answers leaking out). Like cheating on an exam, it makes performance look better than the true ability.

2. The confusion matrix Consider a device in which a warning lamp lights up when the sound-level AI decides "noisy." There are four combinations of the AI's decision and the true state.

This table is called a confusion matrix. In textbooks, a false alarm is called a false positive (saying "yes" when it is actually not), a miss is called a false negative (missing it when it actually is), a hit is called a true positive, and correctly quiet is called a true negative. From this we derive two numbers.

• Sensitivity = hits ÷ (hits + misses): of the time that was actually noisy, what proportion had the lamp on.
• Positive predictive value (PPV) = hits ÷ (hits + false alarms): of the time the lamp was on, what proportion was actually noisy.

Machine learning tools display sensitivity in English as recall and positive predictive value as precision. These are just different names for the same things. They appear in Session 7.

3. Moving the threshold raises one and lowers the other Many AIs output "noisiness" as a number from 0 to 1 and decide "noisy" when it exceeds a certain cutoff (threshold). Lowering the threshold makes the lamp light up more easily: sensitivity goes up, but false alarms increase and the positive predictive value goes down. Raising the threshold does the opposite. One of the numbers alone cannot tell you whether the AI is good or bad. If only "sensitivity 95%" is written, ask for the positive predictive value as well.

4. The pitfall of accuracy Accuracy = (hits + correctly quiet) ÷ total. It is an easy number to understand, but when noisy time makes up only 20% of the whole, even an AI that always answers "quiet" achieves 80% accuracy. Its sensitivity is 0. With imbalanced data, look at sensitivity and positive predictive value, not accuracy.

Hands-on exercise (hand calculation)

A sound-level AI was tried on 1000 seconds of test data.

• The time that was actually noisy totaled 200 seconds.
• The AI decided "noisy" for 100 seconds. Of these, 60 seconds were actually noisy.

1. Fill in the four cells of the confusion matrix.
2. What is the positive predictive value?
3. What is the sensitivity?
4. What is the accuracy?

Review questions

1. If this device were used in a classroom, what problems would arise? Answer separately for "misses" and "false alarms."
2. If you "always want to be notified of noisy time," which way would you move the threshold? What would be sacrificed then?
3. A friend says, "My AI had 92% accuracy." What would you ask in return?

Summary

• Split data into training, validation, and test sets. The unit of splitting is not rows but people or places. If they are mixed, performance looks better than the true ability (leakage).
• Sensitivity = what proportion of the real cases were caught. Positive predictive value = of those called "yes," what proportion were real. Moving the threshold raises one and lowers the other.
• With imbalanced data, accuracy ends up looking good. Look at sensitivity and positive predictive value.

Session 6: What Learning Really Is (Predict → Score → Clear → Analyze → Update) (about 15 minutes)

Goals of this session

1. Explain that "learning" means adjusting the numbers inside a model little by little so that the error becomes smaller.
2. State the five steps of the learning loop in the correct order.
3. Confirm for yourself that swapping the order of one step stops learning without producing any error.

Explanation

Nobody tells the AI the answer, so how does it arrive at the right number?

This time there are only two pairs of data: when x = 2, y = 6; when x = 3, y = 9. The model is prediction = w × x. w is a number inside the model that can be adjusted (a weight, a knob). The answer is w = 3, but we do not tell the program. We start from w = 0 and let it find 3 on its own.

The procedure is a repetition of the following five steps.

1. Predict: make predictions with the current w ((a line of program)).
2. Score: square the differences between the predictions and the correct answers and average them (the mean squared error, the size of the differences expressed as one number). We square them so that positive and negative differences do not cancel each other out.
3. Clear: reset the slope measured last time to 0.
4. Analyze: measure "if w is increased slightly, does the error increase or decrease, and by how much?" This value is called the gradient (slope).
5. Update: move w slightly in the direction opposite to the slope. The coefficient for how much to move it is called the learning rate.

Let us calculate the slope by hand. When w = 0, the predictions are 0 and 0, and the differences are −6 and −9. The slope of the mean squared error is the mean of "2 × (difference) × x": (2×(−6)×2 + 2×(−9)×3) ÷ 2 = (−24 − 54) ÷ 2 = −39. A negative slope means "increasing w decreases the error." So we increase w. With a learning rate of 0.1, w = 0 − 0.1 × (−39) = 3.9. It overshoots right away, but comes back in the next round.

Why is step 3, "Clear," needed? Many machine learning tools are built to add the measured slope onto the previous value (because this is convenient for summing the slopes from many data points). The code this time is built the same way (grad = grad + …). If you do not reset it, the slope from the previous round remains and keeps being added.

No matter how large an AI program is, these five steps are the core of learning. When you read an AI program written by someone else, look for these five first.

New syntax in this session: listed in the file for the AI (.md).

Hands-on exercise

w goes 3.9 → 2.73 → 3.081, repeatedly overshooting and undershooting while the swings shrink, and settles at 3. Nobody told it that the answer was 3.

Exercises to try yourself (try one at a time, and restore the original after each):

1. Change lr = 0.1 to lr = 0.001.
2. Delete line (3) (the grad = 0.0 line that is inside the for loop).
3. Swap the order of line (3) and line (4) (put (4) first and (3) after it).

Before running each one, predict "what will happen," then run it.

(Only for those whose environment can use the machine learning tool PyTorch: the same learning written in PyTorch looks like the following. (3) is optimizer.zero_grad(), (4) is loss.backward(), and (5) is optimizer.step(). In (4), the tool calculates the slope automatically. It often does not run in the browser JupyterLab, so it is fine if you cannot run it.)

Review questions

1. What goes wrong if the learning rate is made too small?
2. What happened when you forgot step (3), Clear? Why?
3. When you swapped (3) and (4), did an error appear? Why did w not move?

Summary

• Learning = repeating Predict → Score → Clear → Analyze → Update, adjusting the weights little by little so that the error becomes smaller.
• The learning rate is the step size. Too small and it never finishes; too large and it overshoots.
• Swapping the order of steps may produce no error. When "it runs but does not get better," suspect the order.

Session 7: Making and Reading a Report Card (Public Handwritten Digits) (about 15 minutes)

Goals of this session

1. Use a public dataset to make a confusion matrix from a model's predictions.
2. Read a confusion matrix as "rows are the true labels, columns are the predictions, the diagonal is the hits," and calculate the positive predictive value and sensitivity by hand.
3. Show with real numbers that overall accuracy hides the classes a model is weak at.

Explanation

We will apply the "yardsticks" from Session 5 to real data.

We use a public dataset of images of handwritten digits. It is included in scikit-learn and can be loaded with load_digits(). It contains 1797 handwritten digits from 0 to 9; each image is 8 × 8 pixels (the dots of an image), and each pixel is a darkness value from 0 to 16.

We split the data into training and test sets, train a simple model on the training set (for practice) (GaussianNB, a model that memorizes "the typical darkness and spread of each pixel" for each digit and answers with the most likely digit), and have it predict on the test set. From the predictions and the true labels, we make a confusion matrix.

Because there are 10 classes, the confusion matrix is a 10 × 10 table. There are only three things to know about reading it.

• Rows are the true labels, columns are the predictions. The "true2" row is the images that were actually 2. The "pred8" column is the images the model answered as 8.
• The diagonal (from top left to bottom right) is the hits. All other cells show how the model was wrong.
• The directions differ: positive predictive value is read down a column (of those answered as 8, how many were really 8). Sensitivity is read across a row (of the real 8s, how many were answered as 8).

classification_report produces, in one table, the precision (positive predictive value), recall (sensitivity), f1-score (a single number balancing the two), and support (number of test images) for each class.

About the data: This dataset is a copy of the test portion of "Optical Recognition of Handwritten Digits" (E. Alpaydin, C. Kaynak, 1998) from the University of California, Irvine machine learning data collection (UCI Machine Learning Repository), and is bundled with scikit-learn. Source: https://archive.ics.uci.edu/dataset/80/optical+recognition+of+handwritten+digits . The terms of use are stated on that page as CC BY 4.0 (free to use as long as the source is credited).

New syntax in this session: listed in the file for the AI (.md).

(If JupyterLab shows ModuleNotFoundError: No module named 'sklearn', create a new cell before that cell and run %pip install scikit-learn once.)

Hands-on exercise

Hand-calculation exercise: Looking only at the confusion matrix, calculate the following. Then check your answers against the numbers in classification_report.

1. The positive predictive value for digit 8 (read down the "pred8" column).
2. The sensitivity for digit 2 (read across the "true2" row).

Review questions

1. Overall accuracy is 0.82. Why can we still say "this model is hard to use for 2 and 9"?
2. Which digit does the model tend to mistake true 9s for? Where in the table can you see this?
3. These digits were written by 13 people. This time we split into training and test sets one image at a time (by row). From the point of view of Session 5, what is the concern?

Summary

• In a confusion matrix, rows are the true labels and columns are the predictions. Hits are always on the diagonal.
• Positive predictive value is read down a column, sensitivity across a row. The directions differ by 90 degrees.
• Overall accuracy hides the weak classes. Look class by class, and also suspect the way the data were split (leakage).

Session 8: Reading the Blueprint of an AI Published by the Laboratory (about 15 minutes)

Goals of this session

1. From the published paper and documentation, draw the laboratory's sleep-scoring AI as a diagram: "input → epochs → features → classifier → output."
2. Calculate for yourself, from numbers given in the paper, the number of points in one epoch (1280) and the number of frequency bins (22).
3. State that the setting values are written not in the main body of the program but in the documentation and the settings file.

Explanation

When you study an AI built by someone else, is the program the first thing you read? The answer is no. The first things to read are the paper and the documentation. The program runs to thousands of lines, but the blueprint fits into one figure and a few paragraphs of the paper.

The subject of Sessions 8-10 is an AI that this laboratory published in 2021 and whose program is publicly available for anyone to use. In the laboratory, it is called "the Tezuka AI" after the first author of the paper (the author whose name is listed first), Dr. Taro Tezuka (affiliated with the University of Tsukuba at the time of publication).

• Paper: Tezuka T, Kumar D, Singh S, Koyanagi I, Naoi T, Sakaguchi M. Real-time, automatic, open-source sleep stage classification system using single EEG for mice. Scientific Reports 11, 11151 (2021). https://doi.org/10.1038/s41598-021-90332-1 (Anyone can read it for free. Terms of use: CC BY 4.0 = free to use as long as the source is credited.)
• Program and documentation: https://github.com/tarotez/sleepstages (Terms of use: MIT License = free to use as long as the copyright notice is retained. The documentation is Readme.md, in English.)

What the AI does. From the electroencephalogram (EEG; the brain's electrical waves) recorded with a single electrode attached to a mouse's head, it decides in real time, every 10 seconds, whether the mouse is currently in wakefulness (awake), non-REM (NREM) sleep (sleep in which slow, large waves appear), or REM sleep (sleep with rapid eye movements; the EEG resembles that during wakefulness). Being able to decide in real time makes experiments such as "shining light only during REM sleep" possible. According to the paper, a single bout of REM sleep in mice is short, about 1 minute on average, and its total amount is only about one fifth of the amount of NREM sleep. The aim of this AI is to catch, in real time and without missing them, these short and scarce periods.

The blueprint (from the text and Figures 1 and 2 of the paper). It has the same form as "cut with a window and decide," which you learned in Sessions 2 and 4.

1. Input: one EEG channel, 128 numbers per second (128 Hz).
2. Epochs: the signal is cut into 10-second segments. These segments (called epochs in the paper) are the same thing as the "windows" in Sessions 2 and 4. In the Tezuka AI, the window is shifted by 10 seconds, the same as its length, so the epochs do not overlap. Because a decision is made when an epoch ends, the paper states that "the delay is 10 seconds."
3. Features (the inputs passed to the classifier): the paper uses three kinds of features that can be computed from any time-series recording: (1) the raw waveform itself; (2) the power at each wave frequency (the spectrum), obtained by dividing 1-12 Hz into 0.5 Hz bins and lining up the power in each bin; and (3) the time of day (how many hours since the lights came on), because sleep is influenced by the daily rhythm.
4. Classifier: a convolutional neural network (CNN; a computation with many stacked layers that picks up shape features from waveforms) followed by fully connected layers; it outputs a probability for each of the three stages and takes the largest as the answer. The paper calls this UTSN.
5. Improved version: the version with an added component that also uses information from past epochs, not only the current one (LSTM; a computation that reads a sequence in order and carries earlier information forward), is called UTSN-L. It uses 10 epochs including the current one. It does not use future epochs (because they have not arrived yet).

Where are the setting values? A tip for reading someone else's program is to look first for the places where numbers are written directly (the settings). In the Tezuka AI, the setting values are listed by name in a settings file (params.json). For example, the lower end of the spectrum bandMin is 1, the upper end bandMax is 12, the bin width binWidth4freqHisto is 0.5, and the classifier type networkType is either simple_cnn (UTSN) or cnn_lstm (UTSN-L). The list of trained classifiers (classifierTypes.csv) has one line per classifier giving "type, points per second, seconds per epoch" (for example, UTSN-L, 128, 10). The documentation also states that UTSN-L does not output a decision until 10 epochs have accumulated (10 including the current one), and outputs "?" in the meantime.

The real program is meant to be installed on a computer together with its tools and does not run inside a browser. So in this session we check only the part that produces the "spectrum" in (2), with a small Python program.

New syntax in this session: listed in the file for the AI (.md).

Hands-on exercise

Hand calculation (on paper first): Using only the numbers given in the paper, calculate the following.

1. How many points are there in one epoch (10 seconds)?
2. If 1-12 Hz is divided into 0.5 Hz bins, how many bins are there?
3. How many seconds from the start of the recording does it take for UTSN-L to output its first decision?

Python: The following cell creates two synthetic 10-second waves (a slow 2 Hz wave and a faster 7 Hz wave, each mixed with random noise), converts them into "power at each wave frequency" using the same epoching as the Tezuka AI, and displays the strongest bin.

In words, the result is: one epoch has 1280 points and there are 22 bins, and even with noise mixed in, the "2.0-2.5 Hz" bin was strongest for the 2 Hz wave and the "7.0-7.5 Hz" bin was strongest for the 7 Hz wave. In the EEG, the slow waves common in NREM sleep (1-4 Hz) and the roughly 6-9 Hz waves common in REM sleep (theta waves) are important clues for distinguishing the stages. The paper chose 1-12 Hz because this range contains both of these bands.

Review questions

1. The Tezuka AI makes one decision every 10 seconds. After REM sleep begins, what is the longest you would have to wait for the decision on the epoch containing that onset?
2. Why is the "time of day" passed to the classifier?
3. Why does UTSN-L output "?" at the beginning of a recording?

Summary

• For someone else's AI, draw the blueprint from the paper and documentation. Read only the necessary parts of the program after that.
• The Tezuka AI cuts a single EEG channel into 10-second epochs and decides the stage from three kinds of features (raw waveform, power at each wave frequency, time of day). The improved version also uses past epochs but does not use future ones.
• The setting values (seconds per epoch, range and width of the spectrum, classifier type) are listed by name in the settings file.

Session 9: Building a Small Tezuka AI and Changing Just One Part (about 15 minutes)

Goals of this session

1. Run a small model based on the same idea as the Tezuka AI on synthetic EEG and read its confusion matrix.
2. Show with numbers how a single change, "also showing past epochs," changes the number of missed REM sleep epochs.
3. Explain why future epochs must not be shown.

Explanation

The real Tezuka AI was trained on 182 six-hour recordings. Reproducing this in a browser is not feasible. So we build a small model based on the same idea. The differences from the real thing are stated clearly.

The synthetic EEG imitates a few properties of the real thing. It moves in the order wakefulness → NREM → (REM →) wakefulness; REM comes after NREM, and each bout lasts about 1 minute on average. NREM has strong slow waves, and REM has strong theta waves. Wakefulness also contains theta waves and has a lot of random noise. So it is built so that wakefulness and REM are easily confused when you look at only 10 seconds.

Logistic regression is a model that multiplies each of the 22 features by a weight, adds them up, and converts the result into probabilities of which stage it is. As in Session 6, the weights are determined by repeating predict → score → update.

The split is made, as you learned in Sessions 5 and 7, by mouse. We use 8 mice for training and 4 different mice for testing. If epochs from the same mouse were in both, the model would get answers right by memorizing that mouse's habits (leakage).

The one place to change is the line PAST = 0. PAST is "in addition to the current epoch, how many epochs back are shown to the classifier." With 0, only the current epoch is shown (the UTSN idea). With 2, the 22 bins of the previous epoch and of the one before it are lined up alongside and passed in (imitating, in the simplest form, the UTSN-L idea of "also using the past"). To make the comparison fair, the first 9 epochs of each mouse are excluded from the scores in every setting (the paper does the same).

The code is fairly long, but you do not need to be able to read every line. What you should read are the PAST line marked with ★ and the output.

New syntax in this session: listed in the file for the AI (.md).

(If you see ModuleNotFoundError: No module named 'sklearn', run %pip install scikit-learn once, as in Session 7.)

Hands-on exercise

First, predict: If PAST is changed from 0 to 2, will the sensitivity for REM sleep (the proportion of real REM that is caught) go up or down? Give a reason in a few words.

Run it: Run the following cell (it may take several tens of seconds in a browser).

The one modification: Change the ★ line to PAST = 2 and run it again.

Laid out in words, the results are as follows (the 4 test mice, 2844 epochs, of which 152 are REM).

Although overall accuracy is high at 0.96, the REM sensitivity was 0.59. What you saw in Session 7, "overall accuracy hides the weak classes," is happening here too. This is because REM makes up only about 5% of the whole.

Review questions

1. With PAST set to 2, the number of inputs the classifier sees at once went from 22 to 66. Why 66?
2. With PAST = 0, 56 of the 152 real REM epochs were answered as "wakefulness." Why is REM easily mistaken for wakefulness? And why did this decrease when past epochs were shown?
3. Showing the next epoch as well might raise performance further. Why, then, does the Tezuka AI not use future epochs?

Summary

• When you cannot reproduce the real thing, build a small model based on the same idea and tabulate the differences from the real thing.
• A single change, "also showing past epochs," raised the model's REM sensitivity from 0.59 to 0.78 and reduced confusion with wakefulness.
• An AI that decides in real time cannot use future epochs. The only input available is the past and the present.

Session 10: Measuring the Effect Correctly and Reading the Table in a Real Paper (about 15 minutes)

Goals of this session

1. State the three things to check before claiming the effect of a modification (make the comparison fair, change the set you measure on, and look at both what improves and what worsens).
2. Re-measure with different test sets and check whether the direction of the effect stays the same.
3. Read from the paper's table which numbers went up and which went down in the improved version of the Tezuka AI, without confusing them with the model's numbers.

Explanation

Last time, with the Session 9 model, the REM sensitivity (the proportion of real REM that the classifier answered as "REM") for the 4 test mice (called "set A" in this session) rose from 0.59 (PAST = 0) to 0.78 (PAST = 2) when past epochs were shown. With PAST = 9 it is 0.91. Can we now say "showing the past makes it better"? The answer is: not yet. In research, we check at least the following three things.

(1) Make the comparison fair. For the two things being compared, use the same test mice and the same epochs counted in the scores. PAST = 9 cannot output decisions for the first 9 epochs, so excluding those from only one side shifts the range being compared. That is why, in the model, the first 9 epochs were excluded in every setting. The paper also excludes the results for the first 9 epochs for all methods compared, because UTSN-L cannot make a decision until 10 epochs have accumulated.

(2) Is the direction the same when the measurement set is changed? A difference found in one test set may be due to chance. In this session, we re-measure on three sets in which the test mice are replaced by 4 different mice. The paper shows the average over three different splits into training and validation data (Table 1), and further confirms the results by testing on 22 separate recordings not used for training and by 10-fold cross-validation (a method that divides the data into 10 parts and uses each part in turn for testing).

(3) Look at both what improves and what worsens. It is common for a single change to raise one number and lower another. In Table 1 of the paper (the average of the three validation runs), comparing UTSN with UTSN-L, which also uses the past, gives the following.

Using the past reduced missed REM (sensitivity went up). On the other hand, the proportion of times that "REM" was really REM when the AI said so went down (positive predictive value went down). In an experiment that "shines light only during REM," sensitivity corresponds to how rarely you fail to shine light during REM, and positive predictive value corresponds to how rarely you shine light when it is not REM. Which one to weigh more heavily depends on the purpose of the experiment. The paper also shows that UTSN tended to detect more stage transitions and to count each bout as shorter.

Do not confuse the model's numbers with the real AI's numbers. In the model, accuracy rose as high as 0.99, and the positive predictive value also went up. The model looks better than the real thing because the differences between stages in the synthetic EEG are clearer than in the real thing. The model is "a tool for checking an idea," and the model's 0.99 is not the performance of the Tezuka AI. The only numbers you can cite as the performance of the Tezuka AI are those written in the paper (in the abstract: overall accuracy of UTSN-L 90%, REM sensitivity 91%, specificity 98%).

Finally, how to check numbers. If you ask a conversational AI, "What is the REM sensitivity in this paper?", it may answer with a different but plausible-sounding number. Always check the numbers of a paper against the tables in the paper. You can also check the table above against Table 1 of the paper.

New syntax in this session: listed in the file for the AI (.md).

Hands-on exercise

Python: After running the Session 9 cell (with PAST = 0 as is), create the following cell below it and run it. It replaces the test mice with three sets and compares PAST = 0 with PAST = 9 (it may take several minutes in a browser).

Laid out in words, the results are as follows.

Hand calculation (reading the paper's table): Look at the "From Table 1 of the paper" table above and answer the following.

1. From UTSN to UTSN-L, list the numbers that went up and the numbers that went down.
2. By how many points did REM sensitivity go up, and by how many points did the positive predictive value go down?

Review questions

1. Even with PAST = 0, REM sensitivity differed greatly: 0.59 in set A and 0.84 in set C. What is different? What does this tell you about the risk of measuring on only one set?
2. In an experiment that "shines light only during REM," which kind of failure would UTSN-L likely reduce compared with UTSN, and which kind would it likely increase?
3. What is wrong with writing "The laboratory's sleep-scoring AI has an accuracy of 99%"?

Summary

• Before claiming the effect of a modification: (1) make the comparison fair, (2) change the measurement set, and (3) look at both what improves and what worsens.
• In Table 1 of the paper, the improved version of the Tezuka AI (UTSN-L, which also uses the past) raised REM sensitivity (87.4 → 90.8) and lowered the positive predictive value (79.2 → 75.2). A single change brings gains and losses at the same time.
• The model's numbers are a tool for checking ideas, not real performance. Check numbers against the tables in the original source.

For those who want to run the real thing: The Tezuka AI's program is meant to run on a computer (Windows or Mac) with Python installed. The documentation (Readme.md) describes, in order, how to install the required tools, how to classify previously recorded EEG (offline.py), and the procedure for training a new model. Errors may occur because of differences in tool versions. In that case, as in Session 1, start by reading the last line of the error.

Criteria for Completion and What to Learn Next

Criteria for completion

If you can explain the following to a friend without looking at anything, you have completed this course.

1. Before passing data to an AI, check the shape of the table and the missing values. When values are missing, an "N-row window" is no longer an "N-second window."
2. Split training, validation, and test data by units such as people or places (mice). If they are mixed, performance looks better than the true ability.
3. Sensitivity is "what proportion of the real cases were caught," and positive predictive value is "of those called 'yes,' what proportion were real." Accuracy alone cannot tell you whether an AI is good or bad.
4. Learning is the repetition of Predict → Score → Clear → Analyze → Update. Even if the order is wrong, there may be no error.
5. In a confusion matrix, rows are the true labels, columns are the predictions, and the diagonal is the hits.
6. Read someone else's AI after first drawing the blueprint "input → epochs → features → classifier → output" from the paper and documentation. Setting values are often stored by name.
7. After changing one part, make the comparison fair, change the measurement set, and look at both what improves and what worsens. An AI that decides in real time cannot use future data.
8. Do not confuse the model's numbers with the real AI's numbers. Check the numbers of a paper against the tables in the paper.

The goal of this course is to read an AI built by someone else, change one part, and check the effect. You can check whether you have reached the goal by trying to change one part yourself and measuring the result. For example, in the Session 9 model, change one part other than PAST (the bin width, the number of mice used for testing, and so on) and measure. Or, in Session 7, replace GaussianNB with another model and measure how the confusion matrix changes.

What to learn next

• This laboratory's papers: The paper you read in Sessions 8-10 is also listed on the "Publications" page of the laboratory website (https://www.med.kobe-u.ac.jp/neurophy/ja/publications.html ). Try reading Figures 1 and 2 (the blueprint) and Table 1 (the performance) using the reading skills you learned in this course.
• Statistical thinking: Methods for stating "there is a difference" or "there is a relationship" while distinguishing it from chance (statistical tests, confidence intervals). This is closely connected to "the numbers move when you change the measurement set" in Session 10. This laboratory's self-study course "The Central Limit Theorem and Statistical Tests" can also be used.
• More on handling tables: The official pandas introduction "10 minutes to pandas" (https://pandas.pydata.org/docs/user_guide/10min.html , in English). It lets you try, with short examples, everything that follows on from Sessions 3-4.
• A book on building neural networks from scratch: Koki Saitoh, Deep Learning from Scratch (O'Reilly Japan, 2016; in Japanese). It lets you build, from scratch in Python, the five steps of learning from Session 6 and the CNN from Session 8.
• What is inside conversational AI: The original source for the "attention" mechanism used by many of today's language AIs is A. Vaswani et al., "Attention Is All You Need" (Advances in Neural Information Processing Systems 30, 2017. https://arxiv.org/abs/1706.03762 ). It contains many equations, so it is best to start with the figures and the introduction.
• Test your skills on public data: There are many public datasets like the handwritten digits in Session 7. When you use one, always check the source and the terms of use (license) on the dataset's description page.

Request for Feedback (Repeated)

We revise this course every time based on the comments of the people who use it. If you have used it, please be sure to send us feedback.

Send to: the address listed on the "Contact" page of the laboratory website Subject: "Course feedback: How AI Works" What we would like you to write: (1) how many sessions you completed (2) what was hard to understand (3) what was good (4) how long one session took

Whether you finished the course or stopped partway, we would very much like to hear from you.

Division of Neurophysiology, Kobe University Graduate School of Medicine

The courses are free for personal study and teaching. Please contact us before distributing modified versions. They are not medical advice.