Weekly update #13 [2/9-8/9]

This is technically my final update. I have finished with the testing and docs. I have also done some refactoring efforts of the phpma code as well as the server code. I still have some things to do but not much. I want the phpma branch to get merged soon.

I refactored the access control in the server. I also refactored the phpma code into a library and fixed failing tests. I added the documentation to all models and components. I left the controllers out for now since most of the functions are very simple. I may add some after the soft deadline. 

As for the error reporting server a few things are left like the server is not currently strong enough against a hostile attacker. I sanitize everything. and cakephp has a good security model but I believe I can make it stronger against hostile input.

I created 2 pull requests but they still haven’t been merged. I will try to have them merged this week. Next week I will be in Germany so I am not sure how much work will I be able to do there at first.

It has been a great summer working on this project. I am happy its almost complete 🙂

Weekly update #12 [26/8-1/9]

This week I have finished writing tests for the error reporting server. I have done tests for controllers and models. I am not going to test views or helpers because it would make the tests very fragile if we change a few html tags or classes. I have 100% coverage on almost all the covered files.

As for the phpmyadmin component. I refactored the functions I wrote into a library that I include. I also renamed functions to follow conventions and wrote the doc blocks for those functions.

For next week I should be documenting the error reporting server codebase. I will probably also add the copyright notice to the required files. There are some refactoring I wanted to do in the access control module. Also I am planning to do something useful to the state like add other states and provide a way to change state.

Weekly update #10 [12/8-18/8]

We are nearing the end of the project. This week was graphs. I created pie charts that showed summaries of the incidents in a report. I also added two more summaries so that almost anything that could be summarizable is summarized.

I had used previously a graphing library but it turned out not to be free for commercial use. Since I am not sure what type of use is this I looked for an opensource library that was more flexible and found one that was released under the MIT license.

I also added syntax highlighting in the stack traces to make it easier to see the code snippets. Furthermore I fixed a bug where if the error occurs in error_report.js file then I wouldn’t accept it. I also added another characteristic for grouping error reports which is pma_version. two incidents belong to the same report only if they have the same pma_version.

I created a pull request for the changes here. I still have not prepared the new seed file. This has a new look for the website and I need a new update from the community on the look of the project. I will be crafting an email once I create the seed file.

As for next week I have planned for adding graphs for the entire website like reports per day as well as frequency of reports per pma_version, as well as other summaries that I may find interesting.

Weekly update #8 [29/7-4/8]

This week is the first week of a two week task to handle related reports and grouping of similar reports. It is also the week that we had our first deployment.

I deployed in order to get more feedback on how everything is going. Sadly the responses were not exactly overwhelming but I have a somewhat good view about how I have decided to proceed.

This week I worked on the js line translator and added url sanitization to most urls in the error report. This helps with two things. First is the increased privacy of the reporters, and second is that there is less difference between similar bugs just because they happened at different hosts.

I have also changed the current schema so that we have reports and incidents. I have moved some of the functionality to the incidents model. I have also changed the views and helpers to handle the new schema. However I faced some problems in how to display certain parts of the incidents. since the incidents do not have a view I need to display the relevant information in the reports view. The most problematic pieces of info is the steps leading to the error and the stacktrace.

They need to be displayed but caution must be taken since they may be 1000s of them. I am currently displaying all the steps at the bottom of the report. As for the stack traces I have not decided how I am going to handle them. I have an idea but I am putting it off incase another idea comes to me. The idea is to have php compare the stack trace to all the stacktraces of the related incidents in the database to find if it is the same as another stack trace of a previous incident, otherwise it is marked in the database as a new stacktrace and is displayed in the reports view.

Another problem is displaying all the different stacktraces. I am not exactly a great designer so I might not be able to create a view that is usable but doesn’t waste a lot of space in the page. We will see about it but I am not sure yet.

I have also yet to handle the manual marking of reports as the same. So for next week I am going to handle that, as well as add a view for the stack traces and maybe even create an html view for incidents and hopefully end up with something better than the last time I tried it.

I will not yet merge my current work into the master branch since it is not finished yet. I will wait till the task is finished to submit a pull request so that the deployment server gets the entire story at once.

Speaking of deployment, I was also able to get the checkstyle errors from around 900 to 43 or 57 depending on who you ask. This was quite impressive. I will hopefully continue on this trend.

Also I might take a couple of days off for Eid at the end of next week. I will try to finish next week’s work before that. I am not exactly behind on my schedule so I don’t think that would be a problem

Weekly update #7 [22/7-28/7]

The task this week was simple. It was to submit bugs to the sourceforge bugtracker. I had already spent a long time trying to get an authentication token in an earlier week. I was able to use this authentication token this week to submit error reports.

Sourceforge uses OAuth V1 which was much more complicated than V2. I was able to use a component created for OAuth V1 for cakephp. I wrote my own component specifically for sourceforge api, similar to the one I did for github. It allows most of the logic to be out from the controller so that it is much simpler.

Since I noticed there is a lack of components relating to apis for cakephp, I wanted to to release them on a separate repo to help others who may need to do what I did. The components I created were a bit specific, not in a way that they can only be used on phpmyadmin or my project, but in that I only implemented the parts of the api I needed. The api is pretty big and while most of the functions are not specific to phpmyadmin, they do not come close to covering a large portion of the api.

Anyway I created code to allow me to get authentication tickets in the future for any application and user credentials so that when the deployment time comes I can easily generate proper credentials other than the ones I was using for testing.

I also finished the line number translator in the phpmyadmin repo. I went with a much cheaper way than what I originally planned. It works great I just have to make it more resilient incase the line count cache file is missing.

Next week is one of the hardest tasks not because of too much coding is required but because I will be dealing with hypotheticals. I am expected to find some sort of measure to group reports together that is as accurate as possible with no way to test it in the wild. There are a million ways it can go wrong and I wouldn’t know until we actually deploy and test it.

I will be trying out the different ways a stacktrace can appear (it changes depending on how the code containing the exception was loaded). I will try to find some way to group them without alienating too many similar reports. We will see how it goes.

Weekly update #6 [15/7-21/7]

This week my task was to create a view for an error report. I finished it successfully. I have created 2 views one that shows the summary of all the submitted reports showing the most important info and the other shows the details of a specific incident

The detailed view is a pure json view that has all the information collected. The reason I kept the view as json is that there was too much information for it to easily fit in a usable interface. When I was doing an html display I was cramming a lot of information and trying to make it look good with the layout. The json saved me from that and allowed me to only include the info that I think would be most suitable and leave the others out to the detailed view. it is also pretty printed so its still easy to manage.

I have tried to make my code as modular as possible so that any change in one part should not impact the other parts. I have left all but the simplest queries out of the controller and implemented them as methods in the model. This is a good approach since it allows me to change the schema of the database and leave the controller and views untouched.

Speaking of changing the schema, the current schema is implemented such that all submitted reports are given first class treatment in the sense that they are listed separately in the index and each has its own view. however this is increasingly not the case, since most likely most of these reports would just be repetitions of an older issue or bug and should not get the same treatment as a new bug or issue.

This is currently implemented by an extra column in the reports table pointing to the id of the report of some similar report. Thus reports without an entry in this column should be given preferential treatment than reports that do.

So I am thinking of changing the schema into two different tables. one for reports and the other for incidents. the incident is a submission that matches a report such that all incidents for a specific reports are for the same bug. and a report has many incidents.

Due to the separation of responsibilities I am able to do the change with minimum changes in the controller and view. I will probably do this in the weeks allotted for grouping reports together since this is the most relevant week and will already include a host of changes to allow for the grouping of reports anyway.

Next week I am going to implement sourceforge api submission. It would allow developers to just automatically submit reports to the sourceforge issue tracker for the developers to work on. The only issue left is access control, in the sense that who can submit these reports?

Weekly update #5 [8/7-14/7]

This week was github api and authentication. I refactored the code I wrote for last week and moved it into a separate component to handle all the requests and try to keep the controller slim. I currently also created some access control on the website where only logged in developers can see the error reports otherwise they are redirected to the home page and asked to login.

After some working I was able to get an access token for the source forge api for my account. I also created a controller to enable me to get an access token for any account since I will be creating an account for sourceforge bug tracker submission. That however is for another week.

For next week I am creating a view for the error report. I am still thinking of how to style the information. I will be using a table with all the info and I will also be changing the way the javascript code sends the data to the error reporting server specifically with the way the stack trace is being sent and nearby context is being sent.