Weekly update #8 [29/7-4/8]

This week is the first week of a two week task to handle related reports and grouping of similar reports. It is also the week that we had our first deployment.

I deployed in order to get more feedback on how everything is going. Sadly the responses were not exactly overwhelming but I have a somewhat good view about how I have decided to proceed.

This week I worked on the js line translator and added url sanitization to most urls in the error report. This helps with two things. First is the increased privacy of the reporters, and second is that there is less difference between similar bugs just because they happened at different hosts.

I have also changed the current schema so that we have reports and incidents. I have moved some of the functionality to the incidents model. I have also changed the views and helpers to handle the new schema. However I faced some problems in how to display certain parts of the incidents. since the incidents do not have a view I need to display the relevant information in the reports view. The most problematic pieces of info is the steps leading to the error and the stacktrace.

They need to be displayed but caution must be taken since they may be 1000s of them. I am currently displaying all the steps at the bottom of the report. As for the stack traces I have not decided how I am going to handle them. I have an idea but I am putting it off incase another idea comes to me. The idea is to have php compare the stack trace to all the stacktraces of the related incidents in the database to find if it is the same as another stack trace of a previous incident, otherwise it is marked in the database as a new stacktrace and is displayed in the reports view.

Another problem is displaying all the different stacktraces. I am not exactly a great designer so I might not be able to create a view that is usable but doesn’t waste a lot of space in the page. We will see about it but I am not sure yet.

I have also yet to handle the manual marking of reports as the same. So for next week I am going to handle that, as well as add a view for the stack traces and maybe even create an html view for incidents and hopefully end up with something better than the last time I tried it.

I will not yet merge my current work into the master branch since it is not finished yet. I will wait till the task is finished to submit a pull request so that the deployment server gets the entire story at once.

Speaking of deployment, I was also able to get the checkstyle errors from around 900 to 43 or 57 depending on who you ask. This was quite impressive. I will hopefully continue on this trend.

Also I might take a couple of days off for Eid at the end of next week. I will try to finish next week’s work before that. I am not exactly behind on my schedule so I don’t think that would be a problem

Weekly update #7 [22/7-28/7]

The task this week was simple. It was to submit bugs to the sourceforge bugtracker. I had already spent a long time trying to get an authentication token in an earlier week. I was able to use this authentication token this week to submit error reports.

Sourceforge uses OAuth V1 which was much more complicated than V2. I was able to use a component created for OAuth V1 for cakephp. I wrote my own component specifically for sourceforge api, similar to the one I did for github. It allows most of the logic to be out from the controller so that it is much simpler.

Since I noticed there is a lack of components relating to apis for cakephp, I wanted to to release them on a separate repo to help others who may need to do what I did. The components I created were a bit specific, not in a way that they can only be used on phpmyadmin or my project, but in that I only implemented the parts of the api I needed. The api is pretty big and while most of the functions are not specific to phpmyadmin, they do not come close to covering a large portion of the api.

Anyway I created code to allow me to get authentication tickets in the future for any application and user credentials so that when the deployment time comes I can easily generate proper credentials other than the ones I was using for testing.

I also finished the line number translator in the phpmyadmin repo. I went with a much cheaper way than what I originally planned. It works great I just have to make it more resilient incase the line count cache file is missing.

Next week is one of the hardest tasks not because of too much coding is required but because I will be dealing with hypotheticals. I am expected to find some sort of measure to group reports together that is as accurate as possible with no way to test it in the wild. There are a million ways it can go wrong and I wouldn’t know until we actually deploy and test it.

I will be trying out the different ways a stacktrace can appear (it changes depending on how the code containing the exception was loaded). I will try to find some way to group them without alienating too many similar reports. We will see how it goes.

Weekly update #6 [15/7-21/7]

This week my task was to create a view for an error report. I finished it successfully. I have created 2 views one that shows the summary of all the submitted reports showing the most important info and the other shows the details of a specific incident

The detailed view is a pure json view that has all the information collected. The reason I kept the view as json is that there was too much information for it to easily fit in a usable interface. When I was doing an html display I was cramming a lot of information and trying to make it look good with the layout. The json saved me from that and allowed me to only include the info that I think would be most suitable and leave the others out to the detailed view. it is also pretty printed so its still easy to manage.

I have tried to make my code as modular as possible so that any change in one part should not impact the other parts. I have left all but the simplest queries out of the controller and implemented them as methods in the model. This is a good approach since it allows me to change the schema of the database and leave the controller and views untouched.

Speaking of changing the schema, the current schema is implemented such that all submitted reports are given first class treatment in the sense that they are listed separately in the index and each has its own view. however this is increasingly not the case, since most likely most of these reports would just be repetitions of an older issue or bug and should not get the same treatment as a new bug or issue.

This is currently implemented by an extra column in the reports table pointing to the id of the report of some similar report. Thus reports without an entry in this column should be given preferential treatment than reports that do.

So I am thinking of changing the schema into two different tables. one for reports and the other for incidents. the incident is a submission that matches a report such that all incidents for a specific reports are for the same bug. and a report has many incidents.

Due to the separation of responsibilities I am able to do the change with minimum changes in the controller and view. I will probably do this in the weeks allotted for grouping reports together since this is the most relevant week and will already include a host of changes to allow for the grouping of reports anyway.

Next week I am going to implement sourceforge api submission. It would allow developers to just automatically submit reports to the sourceforge issue tracker for the developers to work on. The only issue left is access control, in the sense that who can submit these reports?

Weekly update #5 [8/7-14/7]

This week was github api and authentication. I refactored the code I wrote for last week and moved it into a separate component to handle all the requests and try to keep the controller slim. I currently also created some access control on the website where only logged in developers can see the error reports otherwise they are redirected to the home page and asked to login.

After some working I was able to get an access token for the source forge api for my account. I also created a controller to enable me to get an access token for any account since I will be creating an account for sourceforge bug tracker submission. That however is for another week.

For next week I am creating a view for the error report. I am still thinking of how to style the information. I will be using a table with all the info and I will also be changing the way the javascript code sends the data to the error reporting server specifically with the way the stack trace is being sent and nearby context is being sent.

Weekly update #4 [1/7-7/7]

So this week is the first week with the server side component. I am using cakephp as the server software. I was able to get the requests sent by the client side component as well as store them in the database. I found many problems with the sending code of my client side component but after hours days of trying I was able to resolve them. I was also able to use datatables for the listing of the error reports.

Data tables allows for search, filter and sorting of data and I was able to interface it with the server side so that not all the data would be loaded into the page at once since the number of reports should increase rapidly. Currently you can search, sort as well as filter by status and phpma version as well as by error report id.

Currently there is no show page for the error report. That is left for another week. The tasks for next week should have been github integration. I actually started early and finished most of it this week. However it turns out that we may not be using github as planned after all.

The reason I chose github was for two things. I can use it to get user info such as the name and email address so as to forgo the registration page. Also I was thinking of using its issue tracker with the error reports.

However the phpmyadmin developers do not use github’s issue tracker, they use sourceforge. It was suggested that I use that issue tracker rather than github’s. I believe this to be the better choice but as it was not in the plans I wasted some time implementing a github authentication and api component for cakephp. And sadly the way sourceforge does oauth is different from the way github and all the other oauth providers do it so a part of the code cant be used.

But that is not the problem. The problem is that the authentication part of the api is not well documented. I tried it out but it seemed to give me a server 500 error. I was not able to understand why since no relevant error message was displayed to me.

I have submitted a ticket with sourceforge and hopefully they will reply asap so that I can start working on their authentication system. Another thing is that I am not sure whether to offer github integration with sourceforge or only the latter. Github gives me access to things like name and email which allows me to keep logs of the activities done on a report by the developers. It also gives me access credential to check whether the user is a core developer or not by checking if he has commit access on the main repo. This would allow me to do some access control without an admin system to grant users permissions.

but since we would be mainly using the sourceforge issue tracker and there will be nothing to do with a report except to submit it to sourceforge as a bug or mark it as related to another report so that they are grouped together. if there will be no commenting on reports or subscribing to them then emails and names are of no need and we do not need github authentication. however I have still to discuss this with the core phpmyadmin developers

Weekly update #3 [24/6-30/6]

This week I have finished the client side component and once the api between the client and the server is complete I will request merging into master. This should be the last week of me working on the client side component.

My system now currently catches any errors that occur and offer to submit the error report to our servers. It uses file_get_contents or curl depending on availability and respects the proxy settings set by the user.

I wrap all global functions starting with PMA_ I also wrap the callback for AJAX.registerOnload as well as the handler for $.fn.on since it is used excessively throughout the code.

I have tested the code in the latest versions of browsers available on linux (so no IE or safari yet) and it works ok.

I have started on the server side component and created a repo. I have setup the server on my computer. Once I make things more portable I will probably push the initial version of the server early this week. A lot of things are currently dependant on my current setup so it is of no use to push them to a public repo for now.

I will be using cakephp as my development framework. I would have rather used Ruby on Rails but I figure it would be easier on the developers who already know the language to use a php framework.

This week my task is to basically create the server. I have gotten a head start already. I will be using the extra time to take on some of the tasks of the next week so that I would be ahead of schedule.

Weekly update for week #2 [17/6-23/6]

This is the end of my first week of GSoC. I am very happy that I have finished the tasks outlined in this week’s schedule fully. This is the first week of my two weeks in the implementation of the client side component of my error reporting module. So by next week we should be finished with the client side completely and start on the server side.

This week my tasks as per my schedule were:

“create the function that takes an exception and gets all the required info, then anonymizes it. A view is created with the relevant data asking the user to authorize the sending of the report to our servers.”

I have added a handler for the “window.onerror” which collects the necessary info and according to the user settings either sends the error report automatically or ignores it or asks the user what to do. If the user choses to see the error report a modal dialog is opened showing the content of the report as well as asking the user for the steps that lead to the error. Once the user accepts sending the error report the javascript instructs the server to send the error report to our servers.

There is a request being made to the error reporting server but that is not yet implemented so as of yet it should just fail silently. I made sure that all the strings in both the php part and the javascript part are translatable. I have made the javascript respond according to the user preferences like automatic sending or ignoring silently. I have collected a lot of relevant info about the error, however I have not anonymized anything. I did not see a part of the error report that may be considered private however my mentor may suggest a candidate for anonymization once he tries the feature.

For next week my task includes adding of try and catch statements to parts of code so as to get a stacktrace in some cases since this is not possible in “window.onerror”. Also I will try to tackle another problem which is that the line number that is retrieved either in the stacktrace or in “window.error” is for the entire concatenated file of javascripts. This is not very useful as it is not easily reproducible by the developers and is very vague in the sense it is not easy to figure out where this line number is.

When I was writing this report an idea came to me to solve the line number in the concatenated file problem. I can write a bash script that given the get_script.js.php request params and the line number it can output the file which has the relevant number and output the correct relative line number. This is still not very elegant and I will ask the phpmyadmin developers opinions in the mailing list soon.

As for adding try and catch to the functions I have decided that for starters I can wrap them around all functions starting with PMA_. My mentor also suggested I add “AJAX.registerOnload” callback to the list since it is important. I may also wrap other functions manually if I deem them important enough that a large part of the pages uses them or if they do a complicated enough task. but for now it is just the methods I have mentioned. The wrapping is done at run-time so that I will not modify the code of these functions so that this feature can be turned off easily.

I shall do a feature detection to see if a stacktrace is supported by the browser first before trying to wrap these functions in try and catch blocks since it may be useless if the browser doesnt support it. Another problem might be is that if I wait till the document is ready before I do the code wrapping any exception may not be caught by the try and catch blocks however if start too early then the javascript files may not be parsed and some of the functions may have not been defined yet. I still have to research how to do this and what is the best approach.

This is enough for the week. Today I am taking the rest of the day off for a personal matter and I shall start on next week’s tasks tomorrow