2017 School Year Update!

Thanks to my long summer break before I head off to college, I have had the time to make a major update for Academist – version 2.0! Here are all the juicy details:

• When choosing the subject for each of your School Loop courses, there are now specific options for most middle school and high school courses. This makes results more relevant, and is a big improvement over the meager four choices available previously (math, science, history, and economics). As before, Academist uses the course name to make an initial guess of what the subject is during setup.
• Because Academist knows the classes you are in, it prioritizes resources in the corresponding subject areas. For example, if you are learning about energy in chemistry class, a video about chemical energy will carry more weight than one about gravitational potential energy. Links that are in the right subject area for your class will have a checkmark next to them, and links in other subject areas will have a question mark next to them.
• The new "ⓘ Info" button above your Current Assignments will give you useful tips to help you use Academist.
• The settings page now looks much less ugly.
• The math and science textbooks used in the CUSD middle schools are now available as textbook options.
• Other small improvements
• Sadly, in the interest of improving the search algorithm, I haven't had the time to add other study resources besides Khan Academy.

I remember when I thought this blog was a waste of time, but it turned out to be a great place to display information (see the new "An In-Depth Introduction" page). After all, what's a project without a website to tell the world about it? So here's my 2 cents for future geniuses: I highly recommend trying out something other than Blogger. Blogger is incredibly overdue for an update, and there are other free website creators like Weebly that will be easier to use and result in prettier-looking websites. Regardless, try to make your blog into a tool to get people interested in your project. Often times, the English teachers know what they are doing. :)

I also need to address the lasting impact of this project (or lack thereof). Currently, there are 53 users of Academist. However, that number may be skewed, because recent graduates may still have it sitting dormant on their computers. That aside, I am taking a gamble by continuing work on Academist. It has functional limitations that may prevent it from reaching many more people. On the other hand, the staff at CUSD are interested in it and are willing to take the time to spread the word out to the staff & students at the middle schools and high schools. Here's a huge thanks to Megan Arias, Joseph Hartman, and Dr. Rob Nye; without them, I may not have had the motivation to develop version 2.0, and I would not have been able to add the middle school textbooks.

To sum up, Academist is not perfect. A combination of limitations in both Academist and Khan Academy often result in suggestions that are not relevant. Still, they often are relevant and useful. They have to be if Academist is going to help anyone, and that is why I thought it was important to develop version 2.0 in order to increase Academist's chances of success. I have tested Academist and received positive feedback, but I won't claim that everyone will find it useful. Only time will tell.

Final Update

In December, I laid out a plan for what I would do with Academist leading up to April. And what do you know, now it's April! The Genius Project Ted Talk is fast approaching, and it's a good time to review the progress I've made.

I have done everything I wanted to on the plan from December, except for developing a Firefox extension. This was arguably the most ambitious thing I planned for, so it's no surprise I didn't get around to it.

One important aspect of Academist not mentioned before is its ability to work in future years. School Loop may update there website layout and prevent Academist from working, and I had not planned for a way around this until now. If School Loop updates their website next year, I would have to update Academist while in college. I would like to be able to this, so for now, I made it much easier to alter the html-manipulation code if needed. Next year, I will ask a student at Sage Creek to send me the html source of their School Loop page and update Academist accordingly.

The last step for this year will be to present Academist and encourage students to download it in the hopes that some will find it useful.

More Improvements and Future Plans

It's time for an update. Since November 2nd, I've accomplished a few important developments in the slow evolution of Academist. In my last update, I gave myself a small to-do which I've now completed. I wanted to utilize the descriptions of Khan Academy topics in the search algorithm, rather than just the topic titles. To recap, I had a chemistry assignment that was about "colligative properties." This would be a great subject to review in Khan Academy, but Academist found nothing. I searched for "colligative properties" in Khan Academy, and sure enough, there was no page with that title; however, it was in a topic description. Integrating the topic descriptions in Academist was a matter of revising the code that downloads information from Khan Academy, and simply adding that information to the index of topics.

With a an even more "slightly-useful product," as that label still applies in this stage, I decided it would be a good time to add Academist to the Chrome Web Store. This proved not too difficult, and because it is not nearly ready for widespread use yet, I made it so only people with the link can access it, which I might as well put here.

I made a to-do list of the most basic tasks I had to complete on October 4th. I've completed the first to-do, which was developing the searching algorithm and automatically displaying the results under an assignment on School Loop. Now I have finally started on the second to-do: Creating spreadsheets of the table of contents of textbooks at Sage Creek. This is a very time consuming task, but I've already completed a good chunk of the table of contents of my AP Chemistry textbook, which is a relatively long one. Once this first textbook is fully integrated in Academist, I will encourage everyone in my chemistry class (who also uses Google Chrome) to download it, and I will finally start gathering feedback.

Speaking of which, I recently decided to to share the primitive version of Academist with a couple of my friends, only to find out that they both use Firefox. This means I'll definitely be looking into converting Academist to a Firefox Extension. It will be more work, but probably less so than convincing everyone who uses Firefox to switch to Chrome. Unfortunately, there’s not much I can do about those who use Safari and Internet Explorer, but to my knowledge, they are less popular.

Only time will tell if I can accomplish all I want to before the school year ends. That being said, here is a chart of what I plan to do in the coming months:

Small Adjustments for now, but Big Plans for the Future

Since two weeks ago, the constant flow of different assignments in School Loop has been a big help for me in improving Academist, but perhaps also a wake up call for the possible failures that Academist may have in the future.

The School Loop assignments that I have had have forced me to improve Academist's word filtering. Last post, I talked about how I made the Chrome Extension filter out common words in assignments like "study" and "quiz" that would interfere with the search algorithm. I made a list of similar words, including anything that came to mind. However, the words that came to mind were only a fraction of possible words and phrases that should be filtered out of an assignment name. I did not consider the words "lab" or "report" until I had a lab report due. Speaking of which, I also added the word "due" to the list. Here is an example of an assignment for which Academist would not have found Khan Academy topics if it weren't for improving the word filtering:

Academist filters out "Complete" and "Lab" in order to utilize the useful search term "Cons. of Momentum."

For the past two weeks, I have constantly been seeing assignments with words that were interfering with searching for Khan Academy topics. Adding these words has ultimately improved Academist, but the fact that I haven't stopped encountering them is a sign that I may never be able to make the word filtering robust. However, that is only one part of Academist, and certainly not the most important.

Testing Academist against all my assignments has led me to consider another important, yet simple, improvement. Khan Academy contains topics, but also descriptions of those topics. I also noticed that I had an assignment for chemistry class that involved "Colligative Properties," but there was no Khan Academy topic or video with that phrase. However, it was contained in the description of the Khan Academy topic, "Mixtures and solutions." This simply means that I will modify the code that generates the files used by Academist that store data from Khan Academy to include the description for each topic. Adding the description to be used in the search algorithm is also a simple process.

So far, I am still on to-do number one out of five on the to-do list I made on October 4th. There is still a lot to be done, so I will hope that I will have the time to make Academist into a truly useful tool.

A Slightly Useful Product

Last weekend, I was able to spend a lot of time on Academist. I completed the first thing on my to-do list from last post, which was to make a basic search algorithm to find Khan Academy topics and videos based on keywords that may appear in assignments.

The end result is that Academist will successfully display links to relevant Khan Academy pages for an assignment titled "Study for Quiz on Circular/Gravity," and it will know what type of Khan Academy resources they are (It says "Watch" if a video, "Study" if a broader topic, etc.). Luckily, I had this assignment on my list so that I could test Academist against a real assignment, instead of creating fake ones. Here is proof of it working:

Academist will automatically display the top five results using the search algorithm. This all came together thanks to a lot of intricate parts.

First, there was the database of Khan Academy topics. I stuck with only math and science to keep things simple, and I re-created the XML (discussed on 9/19) to include the type of resource for each Khan Academy page. I then had to parse that XML file in JavaScript and translate the data from the file into a JavaScript array.

The next part is the search algorithm. Of course, search algorithms are a widely useful tools that are used all over the place, so I wasn't going to reinvent the wheel. I just had to download a JavaScript library for a search algorithm, and I found one called "lunr." This library allowed me to input a query to search in a list, which in this case was list of all the math and science topics on Khan Academy. It was able to find similar, yet not exact matches, and also ignored common words such as "the." One thing that I did do myself was filter out common words that teachers post in assignments that would get in the way of searching for a topic, such as "study" and "quiz."

Those were the two most important parts, but I also had to learn how to format HTML elements, how to create a hyperlink with JavaScript, how to read local files in a Chrome Extension, and the list goes on. There are endless technical details that are involved in this project, but for the average non-computer-programmer, a picture of the code that makes Academist work might give you the gist of what I am working with:

Building the Chrome Extension - First Steps

After creating a foundation for accessing data for online resources, as discussed in my last post, I decided to start work on making the actual Chrome Extension. I started with the basics, using online tutorials to learn how to setup the file structure and add basic functionality.

One thing that Academist will have to do is be versatile in accessing information on the the School Loop Student Portal, namely assignments. I experimented with sifting through HTML elements and their contents using JavaScript, and I was able to make the program print out the name of an assignment– not too difficult, although it was slow getting used to the JavaScript programming language.

Chrome Extensions are also meant to alter the HTML of a webpage to add new UI features. Of course, Academist needs to do this. The most important feature of Academist will be buttons that are displayed under assignments in the Student Portal, which will open webpages of Khan Academy and possibly other resources related to an assignment. Enabling Academist to be able to take information from an assignment on School Loop and decipher what topic it is on is going to be a lot of work. That will happen later. For now, I'll just add a button that does nothing under every assignment. Currently, the Chrome Extensions displays a button with the text, "CLICK ME" under an assignment whenever the dropdown is activated, as shown below.

I could talk about some technical things now, like how I used a Mutation Observer in order to detect when the dropdown is first opened. Or I could talk about plans for the future, which I think will be more interesting.

Here's a reminder of the two main features I had planned for Academist: Find online resources to help understand topics on School Loop assignments based on keywords in those assignments, and find online resources to help understand topics on School Loop assignments based on textbook page numbers in those assignments and the topics of the sections that those page numbers are in. From my experience, math and science teachers will more often post textbook assignments with page numbers or section numbers rather than explicitly mention the topics that the homework is on. Because of this, I will focus more on the textbook aspect of Academist. But even with that feature, the topics in a textbook still need to be matched with the correct Khan Academy page, and any other online resource I choose to add into the mix.

This leaves some important to-do's:

• Create a basic word-matching algorithm and test it against fake assignments so that it can correctly determine corresponding topics in Khan Academy.
• Convert images of table of contents of textbooks at Sage Creek to useful spreadsheets.
• Convert the spreadsheets to a readable format for the Chrome Extension.
• Create an interface in Academist that prompts student to choose which textbooks they use for their classes.
• Put all of those things together in order to enable Academist to read page numbers in School Loop, then determine the topic name for that section in the textbook, then determine the Khan Academy page that explains that topic.

Starting a Database

The first step I saw fit to take for my project was making a database of topics and review videos for those topics. I started with one of the best homework review sites out there, so you are probably not surprised that I chose Khan Academy.

There is a big gap between the content on Khan Academy and a JavaScript program inside a Chrome extension, so I need to bridge that gap with a database that a JavaScript program can easily parse, AKA read and extract usable data from.

Luckily Khan Academy already has an organized database on their website (khanacademy.org/library), depicted below. Each subject brings you to a page with different topics, and within those topics are subtopics, and then finally are individual videos.

The next step is to create a list of every video in the Khan Academy library, organized into topics. Easy! Just start with individually copying the link to each of the 4,293 math videos...

Just kidding! That would be tedious beyond imagination. Instead I wrote a computer program to put all of the information into a file. I had many different options to choose for accomplishing this, but I decided to use the programming language I am most familiar with, Java (not the same as JavaScript), with a Java editor called Eclipse. Now skip this next sentence if you don't want to get any more technical: I used the Apache HTTP Client to connect to the webpages in the Khan Academy library, along with an HTML parser to extract the data from the website, such as the URL to a video or the title of a topic.

That process takes a while, simply because the program has to connect to the webpage for all 384 topics in the Khan Academy library, not to mention I have to figure out how to solve all the problems with the program before it works correctly (nothing ever goes right the first time with a computer program).

The end result is four XML files, one for each major subject on Khan Academy (math, science, humanities, and economics). XML is just a fancy, but also universally acknowledged data format that can be easily read in most programming languages. Here's what that looks like:

(this is a tiny fraction of the math topics XML file)

One unfortunate factor in the way I chose to work on this part of the project is the fact that the KhanAcademy website is constantly changing. I made this list initially a few months ago, but recently, Khan Academy added more content. That should have been easy to add, but they also changed the naming scheme of the HTML elements that define the topics and their respective URLs. Long story short: I had to rewrite half of the program. It would be crazy to think that won't happen again in the near future, but there's nothing I can do about that. However, there are many different directions this project can take me in, and this is only one small part of it.