Weekly update #15 [16-22/9] FINAL UPDATE

This has been a great GSoC. This was the final week I was working on the project within the GSoC program. I will be doing my evaluations soon. My code in phpMyAdmin has not been merged yet. However the server side code is complete with tests. I am hoping I will be merging the phpMA branch very soon.

This week I have done a major change to the way stacktraces are marked as different among the different incidents, to make it more versatile when reports are grouped. I have also corrected whatever few comments were given on the phpma code.

For the future work, I am going to change the way a part of the phpma code works. I want to output js in the ajax requests rather than json. js gets automatically executed and would simplify the client side js code a lot. However the current implementation of PMA_Response class does not allow js as far as I can tell. I am either going to use ‘echo’ statements or I will have to augment the response class to allow for js responses. I will also talk with my mentor about the best way for me to do this.

I hope that I will do well in my evaluations. I will definitely apply here again next year and hopefully they would have liked my previous work and accept me. This project was very appealing and interesting and allowed me to hone a lot of my web development abilities. It also taught me about designing a good architecture and about the intricacies of cakePHP. I was given freedom in many design choices and I learned from my mistakes. I also created a lot of unit tests. This was my first experience with PHPunit and I learned a lot.

This was my first work in opensource projects but hopefully it wouldn’t be the last. I am going to release at least one of the authentication components I have created as a cakePHP plugin so that no one else needs to reinvent the wheel. I will expand on them a bit and then add some unit tests and release them as a separate plugin. Hopefully this will be the first of many opensource projects that I will create.

Weekly update #14 [9/9-15/9]

I would have loved to say that this week I did alot but to say the truth I havent done much. It didnt help that I didnt find anything important to do, but it was mostly because I was swamped with preparations to my travel to Germany. I am writing this report in Germany right now.

As for the current state of the error reporting project. I believe it is ready to be released. I dont have any more clear ideas about how to make the system better. I have created alot of documentation and the server tests cover almost all of the code. I was actually able to do some refactoring relying on those tests safely.

I am creating another pull request with the latest changes. They are not much as stated earlier. I am going to ask the mailing list about what they think one last time. According to google next week is for docs since the soft deadline is tonight. I already created most of the docs but I think I have a little bit of undocumented code that may need some explanation.

I will be talking to my mentors on where to go from here and whether they think I should be doing something important at this time. But till then I will be getting a care free couple of days to enjoy Berlin. Hopefully I will also get the component merged into phpma/master soon.

Weekly update #13 [2/9-8/9]

This is technically my final update. I have finished with the testing and docs. I have also done some refactoring efforts of the phpma code as well as the server code. I still have some things to do but not much. I want the phpma branch to get merged soon.

I refactored the access control in the server. I also refactored the phpma code into a library and fixed failing tests. I added the documentation to all models and components. I left the controllers out for now since most of the functions are very simple. I may add some after the soft deadline. 

As for the error reporting server a few things are left like the server is not currently strong enough against a hostile attacker. I sanitize everything. and cakephp has a good security model but I believe I can make it stronger against hostile input.

I created 2 pull requests but they still haven’t been merged. I will try to have them merged this week. Next week I will be in Germany so I am not sure how much work will I be able to do there at first.

It has been a great summer working on this project. I am happy its almost complete 🙂

Weekly update #12 [26/8-1/9]

This week I have finished writing tests for the error reporting server. I have done tests for controllers and models. I am not going to test views or helpers because it would make the tests very fragile if we change a few html tags or classes. I have 100% coverage on almost all the covered files.

As for the phpmyadmin component. I refactored the functions I wrote into a library that I include. I also renamed functions to follow conventions and wrote the doc blocks for those functions.

For next week I should be documenting the error reporting server codebase. I will probably also add the copyright notice to the required files. There are some refactoring I wanted to do in the access control module. Also I am planning to do something useful to the state like add other states and provide a way to change state.

Weekly update #11 [19/8-25/8]

This week is the second week for the graphs and stats task. This week was the final week of writing new functionality. Next week is docs and tests. This week I added global stats about the different summarizable facts about incidents as well as an incident chart to track incidents. I also corrected some bugs from last week and added some of the recommended functionality by some of the phpmyadmin developers

I started researching unit testing in cakephp. I have written tests before but that was for ruby on rails. Its the same concept but the available facilities are different. I shall start with writing the tests and I hope to finish all the model functions this week but I am not sure yet.

There is no major changes to the database so the old seed file should still work and no major changes to reports.phpmyadmin.net. I refactored a few parts of the codebase since I didn’t feel them elegant enough. There is probably going to be more refactoring in the coming weeks as well.

I will do some refactoring in the phpmyadmin code I wrote. Since the project is nearly over we shall be merging this code soon so it would be good to be prepared for it.

Weekly update #10 [12/8-18/8]

We are nearing the end of the project. This week was graphs. I created pie charts that showed summaries of the incidents in a report. I also added two more summaries so that almost anything that could be summarizable is summarized.

I had used previously a graphing library but it turned out not to be free for commercial use. Since I am not sure what type of use is this I looked for an opensource library that was more flexible and found one that was released under the MIT license.

I also added syntax highlighting in the stack traces to make it easier to see the code snippets. Furthermore I fixed a bug where if the error occurs in error_report.js file then I wouldn’t accept it. I also added another characteristic for grouping error reports which is pma_version. two incidents belong to the same report only if they have the same pma_version.

I created a pull request for the changes here. I still have not prepared the new seed file. This has a new look for the website and I need a new update from the community on the look of the project. I will be crafting an email once I create the seed file.

As for next week I have planned for adding graphs for the entire website like reports per day as well as frequency of reports per pma_version, as well as other summaries that I may find interesting.

Weekly update #9 [4/8-11/8]

This week completes the two week task of adding functionality of grouping exceptions together manually and automatically. I have changed the way of storing the reports in the database from storing all the data in the reports table to storing exceptions in the incidents table and similar incidents are grouped in the reports table.

I have added functionality to group related reports together such that they show the combined information of their incidents in the reports description page. I have also offloaded the code that creates new incidents and reports to the incidents model. I have refactored it to split it into separate functions.

I have also modified code in phpmadmin to anonymize the host name from the error report as well as remove unneeded params such as database and table names. I have also isolated the actual files and php scripts where the exception occurs to be used in grouping the reports together.

I also added some sanitization in html entities that are stored in the database. there needs to be more sanitization done in the coming weeks to cover all the user submitted data.

I created a pull request here for all the changes including some other checkstyle changes. I will also create a new seed file soon for the new changes since the old one would not work anymore.

As for next week it is the first week in another two week task to add graphs to visualize the statistics both for a single report as well as all the reports as a whole. Additionally I would add another field to the report summary which is which page the exception happens in, as well as sanitize the rest of the user submitted fields to prevent XSS.

Weekly update #8 [29/7-4/8]

This week is the first week of a two week task to handle related reports and grouping of similar reports. It is also the week that we had our first deployment.

I deployed in order to get more feedback on how everything is going. Sadly the responses were not exactly overwhelming but I have a somewhat good view about how I have decided to proceed.

This week I worked on the js line translator and added url sanitization to most urls in the error report. This helps with two things. First is the increased privacy of the reporters, and second is that there is less difference between similar bugs just because they happened at different hosts.

I have also changed the current schema so that we have reports and incidents. I have moved some of the functionality to the incidents model. I have also changed the views and helpers to handle the new schema. However I faced some problems in how to display certain parts of the incidents. since the incidents do not have a view I need to display the relevant information in the reports view. The most problematic pieces of info is the steps leading to the error and the stacktrace.

They need to be displayed but caution must be taken since they may be 1000s of them. I am currently displaying all the steps at the bottom of the report. As for the stack traces I have not decided how I am going to handle them. I have an idea but I am putting it off incase another idea comes to me. The idea is to have php compare the stack trace to all the stacktraces of the related incidents in the database to find if it is the same as another stack trace of a previous incident, otherwise it is marked in the database as a new stacktrace and is displayed in the reports view.

Another problem is displaying all the different stacktraces. I am not exactly a great designer so I might not be able to create a view that is usable but doesn’t waste a lot of space in the page. We will see about it but I am not sure yet.

I have also yet to handle the manual marking of reports as the same. So for next week I am going to handle that, as well as add a view for the stack traces and maybe even create an html view for incidents and hopefully end up with something better than the last time I tried it.

I will not yet merge my current work into the master branch since it is not finished yet. I will wait till the task is finished to submit a pull request so that the deployment server gets the entire story at once.

Speaking of deployment, I was also able to get the checkstyle errors from around 900 to 43 or 57 depending on who you ask. This was quite impressive. I will hopefully continue on this trend.

Also I might take a couple of days off for Eid at the end of next week. I will try to finish next week’s work before that. I am not exactly behind on my schedule so I don’t think that would be a problem

Weekly update #7 [22/7-28/7]

The task this week was simple. It was to submit bugs to the sourceforge bugtracker. I had already spent a long time trying to get an authentication token in an earlier week. I was able to use this authentication token this week to submit error reports.

Sourceforge uses OAuth V1 which was much more complicated than V2. I was able to use a component created for OAuth V1 for cakephp. I wrote my own component specifically for sourceforge api, similar to the one I did for github. It allows most of the logic to be out from the controller so that it is much simpler.

Since I noticed there is a lack of components relating to apis for cakephp, I wanted to to release them on a separate repo to help others who may need to do what I did. The components I created were a bit specific, not in a way that they can only be used on phpmyadmin or my project, but in that I only implemented the parts of the api I needed. The api is pretty big and while most of the functions are not specific to phpmyadmin, they do not come close to covering a large portion of the api.

Anyway I created code to allow me to get authentication tickets in the future for any application and user credentials so that when the deployment time comes I can easily generate proper credentials other than the ones I was using for testing.

I also finished the line number translator in the phpmyadmin repo. I went with a much cheaper way than what I originally planned. It works great I just have to make it more resilient incase the line count cache file is missing.

Next week is one of the hardest tasks not because of too much coding is required but because I will be dealing with hypotheticals. I am expected to find some sort of measure to group reports together that is as accurate as possible with no way to test it in the wild. There are a million ways it can go wrong and I wouldn’t know until we actually deploy and test it.

I will be trying out the different ways a stacktrace can appear (it changes depending on how the code containing the exception was loaded). I will try to find some way to group them without alienating too many similar reports. We will see how it goes.

Weekly update #6 [15/7-21/7]

This week my task was to create a view for an error report. I finished it successfully. I have created 2 views one that shows the summary of all the submitted reports showing the most important info and the other shows the details of a specific incident

The detailed view is a pure json view that has all the information collected. The reason I kept the view as json is that there was too much information for it to easily fit in a usable interface. When I was doing an html display I was cramming a lot of information and trying to make it look good with the layout. The json saved me from that and allowed me to only include the info that I think would be most suitable and leave the others out to the detailed view. it is also pretty printed so its still easy to manage.

I have tried to make my code as modular as possible so that any change in one part should not impact the other parts. I have left all but the simplest queries out of the controller and implemented them as methods in the model. This is a good approach since it allows me to change the schema of the database and leave the controller and views untouched.

Speaking of changing the schema, the current schema is implemented such that all submitted reports are given first class treatment in the sense that they are listed separately in the index and each has its own view. however this is increasingly not the case, since most likely most of these reports would just be repetitions of an older issue or bug and should not get the same treatment as a new bug or issue.

This is currently implemented by an extra column in the reports table pointing to the id of the report of some similar report. Thus reports without an entry in this column should be given preferential treatment than reports that do.

So I am thinking of changing the schema into two different tables. one for reports and the other for incidents. the incident is a submission that matches a report such that all incidents for a specific reports are for the same bug. and a report has many incidents.

Due to the separation of responsibilities I am able to do the change with minimum changes in the controller and view. I will probably do this in the weeks allotted for grouping reports together since this is the most relevant week and will already include a host of changes to allow for the grouping of reports anyway.

Next week I am going to implement sourceforge api submission. It would allow developers to just automatically submit reports to the sourceforge issue tracker for the developers to work on. The only issue left is access control, in the sense that who can submit these reports?