Canvas and AI

We are hearing a lot from faculty about AI, and wanted to let you know how Canvas is dealing with it. First, some refreshing news! They are NOT doing this:

Canvas’s approach to AI has instead been relatively cautious:

1. Opt-in only: in contrast to companies like Google or Microsoft who bundle AI into their products with no opt-out, Canvas is rolling out discrete features that can be turned on or off at either the institution or the course level.
2. Targeted: Canvas is focused on features that could potentially improve faculty quality of life (discussion summaries, smart search).
3. Transparent: Canvas is being clear about what data is sent to the model, what data is used to train the model (not any of our data), and what is logged. They have been publishing AI Nutrition Facts for each feature.

ITG carefully tests new Canvas feature options before enabling them, and the AI-enhanced features are no exception. We evaluate all new features to see if they will be helpful to faculty and students, or if they are more trouble than they are worth.

The following are the results from our exploration into two of these features.

Smart Search

Canvas’s Smart Search uses AI (Cohere Embed Multilingual) to more completely search a course’s content. In contrast with a traditional/simple search, Smart Search understands the context of queries and finds relevant results outside the literal meaning of the search term(s). It facilitates faster content discovery without the need for complex search operators or tools. The feature supports multiple languages and will search content pages, announcements, discussion prompts, and assignment descriptions. It would appear as a menu item in the main left-side course menu:

Smart Search AI Nutrition Facts

Pros:

• Canvas desperately needs a search function, so this is a potentially exciting idea!
• We liked that it supports multiple languages
• It will search pages, announcements, discussion prompts, and assignment descriptions

Cons:

• Like many AI-enhanced searches, this is better as an exploratory tool to find themes rather than a traditional search engine meant to find a particular quoted phrase
• No ability to sort or filter results
• Some bugs (e.g., results from graded discussions show up twice)
• Showstopper: in our testing, students could see information they should not have access to

ITG Verdict: 

We are not turning this on.

Discussion Summaries

The second feature, AI Discussion Summaries, uses a generative AI model (Claude 3 Haiku) to provide instructors with summaries of key points, questions, and ideas in a discussion thread on Canvas. It considers all replies (top-level and threaded) in its summary and is designed to save educators time, especially in large-format classes or other environments where moderating discussions was previously challenging. An important thing to note is that this feature is only available to instructors, students cannot see it at all.

This feature is most helpful for gleaning main ideas and/or the most salient details presented by students in their responses. You can request a summary of the entire discussion in general or provide it with a more specific query or keyword to summarize. In the examples below, you can see how the additional prompt provides even more detailed insight into the discussion content while also verifying the students’ comprehension of the discussion topic.

Discussion Summaries AI Nutrition Facts

Pros:

• We found the AI was providing accurate, helpful information MOST of the time
• Valuable use cases:
  • Verifying comprehension of specific information
  • Gleaning main ideas/most salient details from the replies

Cons:

• It summarized the information on the page, but didn’t accurately determine if the information was correct or not. In at least one instance, it authoritatively said what the correct answer was — and it was wrong
• Since it is a summary, it did poorly when asked for insights about individual responses and outliers to the discussion

ITG Verdict: 

We think this feature would be best for large classes (50+ students) that regularly use discussions, or discussions that have long/detailed replies. Given the occasional inaccuracy, and since most Emerson classes have fewer than 20 students, we are not planning to enable it.

Final Thoughts

As discussed, ITG does not feel like these features are fully ready for deployment at this time, but Emerson does have some exciting projects in progress on this topic. Our AI Working Group is putting together a formalized rubric to help make decisions about the adoption of AI software moving forward—more information to come!