Using AI To Raise Money For Charity

An Important Update

After being in contact with Freerice.com, we decided it would be best to take down the video.

Introduction

There exists this website known as freerice.com. This website is actually really cool and unique! Basically, people can go on this website and answer very simple trivia questions. Each correct response donates the equivelents of 10 grains of rice to the UN World Food Programme. The way this works logistically is after a set period of time, the amount of "rice" that has been raised through the website is matched by sponsors who then make the real world donations to the WFP. You can read more about this in their FAQ. Given how intuitive this website is as well as how simple the questions are, this website is mainly used by children in school as a way to pass the time while doing good. With that being said, this website is open to everyone and personally I think it's a fine way to pass the time.

For todays project though, we are going to go ahead and use this website as a way to practice our programming and ComputerVision. Before we officialy begin though, I want to state that I do not condone or want people to make similar programs. This project is just for entertainment and demonstrative purposes. If you do want to create your own bot, I would strongly recommend not including the autoclicker part. Instead, you could just see if you can create a bot that can give you the answers to the questions but not actually select the answers for you.

With all of that being said, I think we can get started. Also, I created a video talking about this exact project so if that is more your style!

Getting The Data

The first step in this project is of course getting the data. If we take a look at how the website is constructed, we can see that the layout is pretty straightforward. You're given a question and a set of options to choose from.

From here, all we have to do simply select the answer we think is correct. Upon doing so, you your response will either turn green if correct or red if you chose the incorrect answer. There's a bit of a delay to allow the viewer time to process the response before moving on to the next question. The fomrat of the questions don't change and since this is all one application, you never leave the site either. This makes our work a lot easier.

What I can do is use ComputerVision as a way to transfer the information on screen into my Python environment. Doing so is also relatively easy - all that needs to be done is write some code that takes a region in which the question appears. From there, we can use something called PyTesseract to actually read what is being displayed in the screenshot.

We can do the same exact process to have information on what the actual responses are. The only difference is we need to change the region in which the screenshot is taken. Most of the time, there will be four options (I've noticed that sometimes only three options appear but that's not a big deal), so what I did was have take a screenshot of the four locations where the options appear. I chose to do individual screenshots for each option because now I'm able to associate each region as it's own element which will make the answer selection process much easier down the line.

Answering The Questions

So now we have a way to read the text on screen, next up is finding a way to actually answer the questions. For the math problems, it's relatively straightforward. When we use PyTesseract, we're given a string back. From here what we can do is search for the "=" sign that appears on every question. Using that sign, we can split the question up using .split(). If we take the first index of the split results, we're left with the mathematical equation which Python can very easily solve. In the picture above, Python would simply type 8x6 which would of course return 48

While that's pretty straightforward, I wanted something that would give me 100% accuracy. Why wouldn't the previous example work every time? The way the site works, as you progress through the questions, they slowly get more and more difficult. The equations sometimes become pretty complex and there are a lot more symbols involved in the questions. While Python would have no problem figuring out the solutions, the issue lies with the way we get the information -- PyTesseract. PyTesseract is fantastic at what it does but I noticed that on a few occassions, it would have a bit of trouble copying down some text correctly. Any imperfection would cause the code to break. It is for this reason that I decided to employ a new strategy. Instead of doing math problems, I decided to take a look at the categories to try to see if there was something else that I could do that would make me feel a bit more confident in my code.

It was at this moment that I found the category, US State Capitals.



If we take a look at the new questions, it might be a bit counterintuitive as to why we would choose something like this, instead of trying to modify my code and continue to do simple math problems. Simply put, there are only 50 different possible questions that can be found in this category. Working with only 50 questions makes code adjustment much easier. Luckily for me, my code was able to read the question correctly every single time for all 50 outcomes. The next issue is how do we solve these questions. It's not like I can just type them into the Python console and get an answer -- especailly offline. This fix is also pretty easy. What I did was I simply went online and I copied a table that had every state and their capital listed together. Turning that into a .csv file, I was able to very easily import the new table into Python.

From here, I used some similar logic as before. I used the word "is" as a way to split the question up into two different parts. I then chose the first index and that gave me the name of just the capital. In the example above, that means that after splitting the text up, I'm simply left with Santa Fe. From here, all we have to do is match up Santa Fe with the table index that we imported. Once we find a match, we can easily see that Santa Fe is associated with New Mexico, thus, giving us the answer.

Selecting The Answer

This part was by far the easiest and took only a few seconds to put together. If we recall, I created individual screenshots for each of the regions where an option can be found. Using PyTesseract, I can store the text from each response into it's own element labeled Option 1-4. I then iterate through each option until I find the option that coincides with the answer that we concluded earlier. In this case, that would be Option 1. I can then just write some code to leftclick the region where Option 1 is stored and... that's it. 

def clickAnswer(value):

	if value == 0:
		pyautogui.leftClick(950, 300)
	if value == 1:
		pyautogui.leftClick(950, 360)
	if value == 2:
		pyautogui.leftClick(950, 420)
	if value == 3:
		pyautogui.leftClick(950, 480)
After this, the only that that's left to do is add a tiny bit of delay to take into account the time it takes to transition between questions, that's it! You can have this running for as long as you would like. In the video, I had the code running for a little over 2 hours and I was able to raise over 15,000 grains of rice.

Conclusion

While this project wasn't very difficult, I still think it's a good beginner project to get people interested in ComputerVision. As mentioned before and in the video, I do not recommend someone recreating this bot because that defeats the purpose of the website and takes the fun out of everything. As an alternative, perhaps people can create a bot that tries to answer the questions from this website but don't do the last part of automating it. If you do want to see it in action though, I recommend watching the video I created, the last few minutes is a time lapse of the bot answering questions! With that being said, go check out the freerice.com! It's a really fun website and a fun way to kill some time while also doing good.