Developing Critical Thinking

in the Era of ChatGPT

Alex Avdiushenko

aleksandr.avdiushenko@jetbrains.com

To make a long story short

Oral Tradition
(circa 100,000 BCE - 3,200 BCE)

Birth of Writing and Numeracy
(circa 3,200 BCE)

Scrolls and Libraries
(circa 1200 BCE — 15th Century CE)

The Great Library of Alexandria in Egypt

ALEXANDRINA BYBLIOTHECE

The Printing Press
(1440 CE)

Verification problem

Malleus Maleficarum (Hexenhammer) — Hammer of Witches

Probably the best known unscientific treatise, which was very popular at the time..

The more accessible information becomes, the more verification we need.
This issue is as old as our history itself. HERE you see the Malleus Maleficarum, also known as the "Hammer of Witches". This book is a stark example of the lack of scientific validation methods that existed during its publication. Originated in 1487, the book was an unscientific guide used during the period of the witch trials in Europe which established the justice framework for witchcraft.
This book was accepted largely based on social beliefs rather than empirical evidence. Today, it is even more important not to fall prey to unfounded assertions. After all, we certainly don't want to fuel our decisions with a "Hammer of Witches".

Electricity and Telecommunication
(19th century)

Postal delivery time from London in 1850

days

to reach

New York

Alexandria, Egypt

Istanbul, Turkey

Mumbai, India

Singapore

Shanghai, China

Sydney, Australia

Digital Age and the Internet
(late 20th century)

Artificial Intelligence and ChatGPT
(21st century)

The fact is that nowadays any information is extremely accessible! Even without an AI assistant, one has access to a massive amount of information.
For example, on YouTube, you can find an explanation for almost anything and to solve almost any problem, from repairing not working washing machine to executing the proper professional technique for a squat exercise, as we see here on the slide.
Moreover, often, you can not only watch the video, but interact with the information, change some features and learn by exploration. With modern tools, you can even create similar materials yourself.
For instance, using Universe Sandbox application, you can literally make planet Mars collide with planet Venus and see the result. This simulates modern scientific theories about the world and displays all the physics of the process on your screen within a few minutes. Moreover, I am sure there is already, or definitely soon will be, a similar application for any scientific area.

Ok, and we go on!
And the next huge thing is that modern AI assistants not only search for facts and information, but they also combine and even generate it successfully. We might even say that they generate a kind of new, original content. Partly, it is true even for these slides.
So we get an enormous amount of information, and the challenge is that we don't know at all whether it is true. Here we encounter a fundamental problem: is it possible to somehow verify the answers of assistant, that might be smarter than we are =) Or maybe we can at least align it with our goals and work together effectively? In fact this question is most relevant for education, how we can use modern technologies safely and for the benefit there?

The Influence of ChatGPT in Education

Chegg Inc — an NYSE-listed company focused on improving learning and learning outcomes for students

Now let's move on to the second part of the talk and let's discuss the influence of ChatGPT in education.
We start with the example of CHEGG company, which focused on improving learning and learning outcomes for students. This company, as many others, was hit hard by the launch of ChatGPT.
In May 2023, the company said, “In the FIRST PART of the year, we saw no noticeable impact from ChatGPT on our new account growth, and we were meeting expectations on new sign-ups.”
But students that paid Chegg to practice exams and get homework feedback turned to ChatGPT instead. As a result, the company share price decreased more than 40%. In August 2023 the company already said, “We’ve pivoted the company to harness AI to better serve learners.” Now they are building internal LLM in partnership with Scale AI.

Student's cases of using AI assistance

One of the exams had the format “there is a realistic case study, you need to describe the actions necessary to solve it.” We had only two samples from the teacher: 2 situations and 2 expected answers to them. So we added the theory from the slides of all the lectures into the GPT, followed it with exam samples and asked it to generate similar tasks and answers to them based on the received theory. Super fast preparation for this kind of exam))

The process of completing practical homework with writing code has now changed: ChatGPT writes basic, often broken, code, which I take as the basis for further execution of the task.
In the further process of execution, Yaroslav uses it as a bug fixer — it is not always easy to understand what the error is. So you can throw the code into it and ask to find and fix the error.

And of course, students master new technologies just as well as you and I or even better.
Consider two typical examples from students of our educational program in Cyprus. NADEZHDA with her friends had one exam in the format of solving practical cases. They had to prepare using only two task samples from the teacher. So they added the theory from the slides of all the lectures into the GPT, followed it with exam samples. And asked AI to generate similar samples based on the received theory. It was a superfast preparation for this kind of exam!
Let's look at another example from YAROSLAV. He was given some task, to write some kind of convolutional neural network. Initially, when looking at such a task, it is not always clear how to approach it and where to start. In the past, Google solved this, helping with existing solutions and showing how related tasks are done.
Now ChatGPT just writes the first version of code. Usually, approximately 20% of this code remains in the final version of neural network; sometimes it writes completely inadequate code, which, however, helps to understand what to Google next. In the following process of execution, Yaroslav uses ChatGPT as a bug fixer.

Pros and cons of ChatGPT in studying

Useful cases

helps answer questions for tools with poor or unreadable documentation (for example, it can tell you how to start using GDB)

gives good and quick examples for some JSONs, XMLs with the necessary characteristics, or some fixtures to fill in the database

helps to overcome the fear of a “blank slate”

What it doesn't handle well

any task where you need to understand deeply the general context of the whole project

not typical problems in not so popular languages: if you ask to solve a dynamic programming problem in Haskell, it gives funny answers

for different mathematical facts a lot of the GPT-generated proofs are incorrect

So what are the pros and cons of using ChatGPT in studying?
Let's begin with the USEFUL CASES.
It HELPS answer questions for tools with poor or inaccessible documentation (for example, it can tell you how to start using GNU Debugger).
It PROVIDES nice and quick examples for some JSONs and other objects, or some values to fill in the database.
ONE of the main points: it helps to overcome the fear of a “blank slate.”
And WHAT it doesn't handle well:
Definitely it is ANY TASK where you have a large general context of the whole project.
ALSO, it is non-typical problems in not so popular areas: if you ask to solve a dynamic programming problem in Haskell, it gives funny answers, which are totally wrong.
CRUCIAL for us: for different mathematical facts a lot of the GPT-generated proofs are incorrect.

Will AI replace me?

nsk_illustration

2003, silver medal pms

2009, master of math mmf

2014, Ph.D. ict

2016, YDS shad

2015-2018, analyst
yandex

2016-2022, ed manager
csc

2023-..., researcher

GPT-3

GPT-4v

pay_the_bills

Let's switch gears and dive into my personal story and also let's try to think, if AI replace me?
I was BORN and raised in faraway snowy Novosibirsk in a beautiful Soviet family: mother is a doctor, father is an engineer, also I have an older brother.
Childhood in the 90s was fun: garages and unfinished buildings as playground, the sea and the forest instilled a love of sports and tourism. In school, about the fifth grade I tried mathematics, already in the seventh grade I won the regional math olympiad. It was almost a miracle. Away we go — a physics and mathematics school in the legendary AKADEMGORODOK, close to Obges, which is microdistrict. Then MATH at Novosibirsk State University, science... The MOTTO was "You could not be a scientist, but a candidate of science must be!” =) So I made my PhD thesis in math modeling there in the Institute of Computational Techonlogies.
In graduate school, I wanted to try something else. In parallel, the hype about machine learning grew; a YANDEX company wanted to come to Novosibirsk with the development office and Yandex Data School. This chance could not be missed, and I was fortunate to simultaneously become a curator and student of the Yandex School Novosibirsk branch. It was truly challenging: I even took academic leave for the first time in my life. Before my graduation in the fourth semester, we had a daughter in our family, but I liked everything and was happy :) I already worked as a data scientist at YANDEX for the graduation from Yandex Data School.
It was super-useful for about three years of experience in industrial programming and analytics, but I became increasingly aware that I was drawn back to teaching, science and organizing education.
So I moved to Saint Petersburg and started working as a HEAD of the machine learning bachelor program in the new department of mathematics and computer science of St. Petersburg State University. It was such an educational startup within the university — a very strong team, support from Yandex, JetBrains, Gazprom Neft and other companies, many ambitious tasks and one big goal to create something like MIT or even better in Russia.
Now I am working here at JetBrains, developing a new bachelor program "Computer Science and Artificial Intelligence" with Neapolis University.
So now goes an intriguing question, can or when does AI replace me?
The researchers tested GPT-3.5, the previous best model from OpenAI, on reasoning problems found in intelligence tests and exams like the SAT. Notably, SAT is a standardized test used for college admissions in the United States. It is a multiple-choice, pencil-and-paper test that measures a student's reading, writing, and math skills. Surprisingly, GPT-3.5 performed impressively well, solving 75% of the problems in SAT Math correctly. This outperformed the average score of human participants.
Notably, GPT-4v performed even better on SATs. It scored 700 out of 800 points on the math SAT. Personally, I tested GPT-4v on some assignments of Yandex Data School, and it solved a lot of them successfully.
Anyway, I can still DRIVE a car, understand people, and hope that it will help me pay the bills.

University teachers

Do you use AI assistant in your everyday practice?

What are the reasons for not using AI assistant?

Intentionally, so as not to accidentally violate copyright

AI is not reliable yet. If you believe it, damage may occur in almost everything related to information security, protocol development, infrastructure, etc.

C++ requires human verification =)

ChatGPT has problems with generating correct mathematical proofs

You may ask why not using AI assistant, what is wrong with them? Let's DELVE into understanding what the issues are.
FIRSTLY, there is a deliberate attempt to avoid copyright violation. This is an important consideration in today’s digital age.
NEXT, we must acknowledge that AI technology has not yet reached a point of complete reliability. This is crucial because over reliance on AI without proper verification can lead to significant issues in information security, protocol development, infrastructure, and other related domains.
THIRDLY, when it comes to programming languages such as C++ or Rust, human oversight is still required.
LASTLY, there are specific challenges that ChatGPT faces, such as difficulties in dealing with mathematical proofs.

Demo: modern tools for teachers

PyCharm with TeXiFy IDEA plugin — LaTeX, Python and JavaScript in one place with great autocompletion, inverse search from PDF, git etc.

AI assistant helps with translations in a broad sense: not only between languages, but also any formats, short ideas to nice descriptions, text to speech, auto tests creation, etc.

GitHub Copilot plugin offers very reasonable pieces of code, draft solutions, and generally helps speed up writing text content

Grazie Pro plugin corrects typos and polishes long sentences

Let me switch gears second time and share my own workflow for teaching and lecture preparation.
FIRSTLY, I usually work in PyCharm complemented with the TeXiFy IDEA plugin. This combination brings together LaTeX, Python, and JavaScript all in one place. It allows using powerful autocompletion and inspections of PyCharm, and inverse search from pdf.
NEXT, let's discuss the AI Assistant. It is capable of assisting with translations in a broader sense. AI helper is not just restricted to translating languages; it can convert formats, elaborate short ideas into detailed descriptions, and even to oral speech, facilitate auto tests creation, and more.
THIRDLY, I'm using the GitHub Copilot plugin. It offers practical pieces of code, draft solutions, and generally helps speed up the writing of text content for my instructions or lectures.
LAST but not least, the Grazie Pro plugin checks for typos and polishes long sentences. Its purpose is to ensure your text is clean, concise, and clear before reaching your audience.

Here I'd like to show the concrete example from my own experience. It is producing the problem for the International Mathematical Competition.
First of all, I have chosen an appropriate problem from ancient times of my teaching at Novosibirsk State University. There I organized math contests with tricky olimpiad tasks. So I need to translate the problem from Russian to English and to make a Latex format from Microsoft Word.
You see, with the help of AI assistant; I can do it in a few seconds. Here I simply copy-paste the answer to the working area. Then I check the result and look at the Grazie suggestions. Here I don't accept it, because it suggests something strange about additional dot in the text after the word "elements", but often it helps to polish the text.
Then I compile Tex sources and look at PDF file.
Then I need to write a solution, so I think through the solution and type it in Latex format. So firstly, I write x^2 = 0, then latex Rightarrow command goes. Here you can see that GitHub Copilot helps me to write the solution very quickly; it even makes simple calculation with math formulas.
Then I write the second part of the solution. Here I need to prove that the product being equal to zero is commutative. So that ab=0 if and only if ba=0. You see, sometimes I disagree with the Copilot's suggestions, but often it helps. Eventually, I check everything, recompile Tex sources and get the final PDF file done. Really fast, effective and convenient workflow, at least for me. Maybe you will consider using something similar for your own practice.

Few observations from the Academy about studying

Incredible speeds: any new knowledge is immediately available to everybody

Yesterday we found out smth on Twitter $\to$ tomorrow added a new skill to ourselves $\to$ in a week all the students have it

Therefore solving unsolved problems is the key skill

Essential activities for critical thinking development are

Mathematical proofs

Debugging

System engineering

Human discussions

Let's go on to a few observations about modern education from our teachers.

They NOTED the incredible speed of learning today. Essentially, as soon as new knowledge emerges, it becomes almost immediately available to everyone. This speed makes a significant impact.
For instance, IMAGINE we discover something on Twitter today. By tomorrow, we've already added a new skill to our repertoire. In just a week, all of our students have acquired this skill.
That's why the CRUCIAL skill nowadays becomes the ability to solve unsolved problems.
ALSO our teachers highlighted certain essential activities for the development of critical thinking.
Those activities include crafting mathematical PROOFS,
DEBUGGING problems in code or systems,
tasks related to system ENGINEERING, including robotics,
and, importantly, engaging in HUMAN discussions and debates.
Each activity, in its own way, sharpens our ability to evaluate information critically and make reasoned judgments.

Another four AI-powered cases from teachers

Create lecture outline $\to$ Make slides in reveal.js (click on the links!)

Checking how easily an assignment task can be answered with the help of LLM

DALL-E 3 draws beautiful high-quality illustrations, in raster or in vector formats

Send all routine tasks in a broad sense to the ChatGPT (e.g., writing syllabus draft)

Here in the final part of the talk, let's look at the practical use-cases of AI that teachers are utilizing right now.
FIRSTLY, AI can significantly streamline the creation of lecture materials. With a simple prompt, AI can help you generate a detailed lecture outline and even convert that outline into visually engaging slides using tools like reveal.js. This is a great example of how AI can facilitate the preparation process for educators.
NEXT, teachers are experimenting with AI's capabilities by checking the effectiveness of LLM in answering assignment questions. This not only tests the AI's understanding but also helps to identify how students might use such technology to aid their learning.
In the realm of ART and visuals, DALL-E 3 is another remarkable tool. It's capable of creating nice, high-quality illustrations, whether in raster or in vector formats. These illustrations can then be used to enhance teaching materials and engage students.
LASTLY, AI is taking on the task of handling routine responsibilities. From drafting course syllables to managing other repetitive tasks, AI is opening up space for teachers to focus on more personalized and creative aspects of their work.

Instead of a conclusion

Reliable information sources have evolved, but the need for critical thinking and verification remains constant or even more important

Incredible speeds give us incredible opportunities

JetBrains has an AI platform for everyone, and this is cool!

We strive to pioneer in integrating AI in education tools also, shaping a smarter, more efficient learning environment

Come teach with us — it's fun and gives great development

🥳🎉🎄 Text me anything! 🎄🎉🥳

As we reach the final of the presentation, I'd like to leave you with some takeaways.
FIRSTLY, the panorama of reliable information sources has evolved over the centuries. From the era of oral traditions to the digital age of AI, we have witnessed a monumental shift. Yet one constant remains — the need for critical thinking and verification. In today's fast-paced world, this need is not just constant but has become more crucial than ever.
SECONDLY, the incredible speed at which information is now accessible presents us with new opportunities. It allows for rapid learning, immediate access to knowledge, and the ability to connect globally. However, it also demands from us a heightened sense of discernment and responsibility.
As a COMPANY, addressing these changes means adapting and innovating. We already have 'JetBrains AI platform', integrated across all the products, improving our workflows and decision-making processes.
FINALLY, we don't just adapt but lead. We have the opportunity to be pioneers in integrating AI into education tools. By doing this, we can shape a smarter, more efficient learning environment.
So, I invite you to JOIN us in teaching and in development new learning environment; it's really fun and exciting.
Your thoughts, FEEDBACK, and ideas are invaluable. I encourage you to connect with me on Telegram or GitHub. Let's collaborate and shape the future of education together. Thank you.
Happy New Year and Merry Christmas!