ECE497 Project Sudoku Solver

Team members: Joey Pierce, Donglai Guo

Grading Template
I'm using the following template to grade. Each slot is 10 points. 0 = Missing, 5=OK, 10=Wow!

 00 Executive Summary 00 Installation Instructions 00 User Instructions 00 Highlights 00 Theory of Operation 00 Work Breakdown 00 Future Work 00 Conclusions 00 Demo 00 Late Comments: I'm looking forward to seeing this.

Score: 10/100

(Inline Comment)

Executive Summary
Our project is about implementing a sudoku solver on the BBB. We will first feed the picture of the unsolved 9x9 sudoku grid to the BeagleBone and display it onto the LCD. We are planning on using Google vision to convert the text (numbers) of the picture into a text file. Then we press a GPIO button to trigger the sudoku solver which takes a text file as an input and outputs the solved sudoku grid. The correct result will be shown on the LCD too.

What works: Right now we are using a PlayStation camera to take a picture of our Sudoku board that we send to the Google Vision API. We are now able to use a board with grid lines, but still need to put in zeros for the blank puzzle spots. We are using ImageMagick features to remove the grid lines on the board before sending it to API. We then take the API results and send it to our Sudoku solver. We can then take the final result and use ImageMagick to compose a blank grid onto the Sudoku answers to display a completed puzzle on the LCD Display. Our Sudoku solver came from GitHub: https://github.com/Sanahm/SudoCAM-Ku.

What doesn't work: Our main problem right now is increasing the accuracy of the Google Vision API. Since we are attempting to remove the grid lines with ImageMagick, we found that its a little harder to get a good quality picture that the API can easily detect.

Hardware Setup



 * Connect the SPI and GPIO jumper wires of the LCD screen to S1.1 and GP0 on BBB.
 * Connect GP1 jumper wires to a random spot on the breadboard.
 * Connect GP1_3 (white wire, also third wire from right to left) to one side of a pushbutton.
 * Connect the other side of the button to ground.

Installation Instructions
The installation of this project is as follows:

First you need to set up the Google Vision API. root@beagle$ git clone https://github.com/pierceja/BeagleboneSudoku.git

root@beagle$ git clone https://github.com/pierceja/BeagleboneSudoku.git

User Instructions
Once everything is installed, how do you use the program? Give details here, so if you have a long user manual, link to it here.

Once everything is installed and set up, you can execute the capture.sh script by doing: ./capture.sh When this begins to run, the camera will begin capturing frames which will be displayed on the LCD display. You need to capture a clear frame of the Sudoku puzzle you are trying to solve. The challenging part is capturing a frame which shows every row and column clearly. Once you see a frame you like on the LCD display, quickly press CTRL+C on your keyboard. This will initiate the image processing functions on the frame. Once image processing is complete, the processed image will be displayed on the display. If you are satisfied with the image, you can press the GPIO push button to send it to the Google API. The detected digits will be shown on the LCD display. Once this occurs, you can press the push button again to send the digits to the solver. After the solver is complete, the final results will be shown on the display.

Highlights
Here is where you brag about what your project can do.

Our project can solve a Sudoku puzzle from a single frame. It uses some cool image processing features from ImageMagick to remove the grid lines of the Sudoku puzzle and can create a new puzzle grid to display the results. It is also pretty user friendly and just requires a couple GPIO button presses after the frame is captured. Below is video showcasing our project:

Include a YouTube demo.



Theory of Operation


The flow of the program can be described as follows:

1. The program starts with a while loop that keeps taking photos until user presses "Ctrl+C" which will capture the image;

2. The image is then processed using ImageMagick to remove grid lines in order to achieve better results; Both orignial image and processed image will be seen on the LCD.

3. Google API will do Digits Recognition on the processed Image and send all numbers in a Json file back to the bone;

4. The program then will convert the json file into a text file from which Java program will get input. The Image from google will also be shown on the LCD;

5. Java program has two parts: The first part is correction.java which is written in an attempt to fixed the distortions from Google API(such as spaces, brackets and etc) and make it compatible with the actual solver. The second part is Sukoku.java which is basically an algorithm that will be used to solve the Sudoku. The correction.java actually have 2 versions, one is to remove distortions and the other one is to append 0s in blank spaces. But the later one seems unnecessary because we were not able to make it work with blank grids;

6. Finally, ImageMagick techniques are used to composite two picture (one from soluton.txt and the other one from a boundbox.png) and also final image will be shown on LCD;

Work Breakdown
List the major tasks in your project and who did what.


 * Google Vision API setup-Joey
 * Hardware interfacing-Joey
 * API and solver merging-Joey and Donglai
 * Solver setup and editing-Donglai
 * Image processing work-Donglai

Also list here what doesn't work yet and when you think it will be finished and who is finishing it.

As mentioned above, right now the API is completely accurate. This is being worked on by both of us and should by fixed by Tuesday November 14, 2017.

Future Work
Suggest addition things that could be done with this project.

There are a couple of neat ideas that we came up with that could be added to this project. One idea would be to get a stream from the camera so that it's a bit easier to capture a good frame. Another idea might be using image processing to append zeros to the Sudoku puzzle so that you didn't have to draw them by hand.

Conclusions
Give some concluding thoughts about the project. Suggest some future additions that could make it even more interesting.

Overall, this was an interesting project to tackle. The main challenge of this project was dealing with aspects that we had little control over such as the API. It was interesting to work with an API, however, and to experiment with trying to get the best accuracy we could out of it. Another challenge was making the most out of the hardware we were given. I think we might have gotten better results if we had a high end camera to use.