Weekly update #15 [16-22/9] FINAL UPDATE

This has been a great GSoC. This was the final week I was working on the project within the GSoC program. I will be doing my evaluations soon. My code in phpMyAdmin has not been merged yet. However the server side code is complete with tests. I am hoping I will be merging the phpMA branch very soon.

This week I have done a major change to the way stacktraces are marked as different among the different incidents, to make it more versatile when reports are grouped. I have also corrected whatever few comments were given on the phpma code.

For the future work, I am going to change the way a part of the phpma code works. I want to output js in the ajax requests rather than json. js gets automatically executed and would simplify the client side js code a lot. However the current implementation of PMA_Response class does not allow js as far as I can tell. I am either going to use ‘echo’ statements or I will have to augment the response class to allow for js responses. I will also talk with my mentor about the best way for me to do this.

I hope that I will do well in my evaluations. I will definitely apply here again next year and hopefully they would have liked my previous work and accept me. This project was very appealing and interesting and allowed me to hone a lot of my web development abilities. It also taught me about designing a good architecture and about the intricacies of cakePHP. I was given freedom in many design choices and I learned from my mistakes. I also created a lot of unit tests. This was my first experience with PHPunit and I learned a lot.

This was my first work in opensource projects but hopefully it wouldn’t be the last. I am going to release at least one of the authentication components I have created as a cakePHP plugin so that no one else needs to reinvent the wheel. I will expand on them a bit and then add some unit tests and release them as a separate plugin. Hopefully this will be the first of many opensource projects that I will create.


Weekly update #14 [9/9-15/9]

I would have loved to say that this week I did alot but to say the truth I havent done much. It didnt help that I didnt find anything important to do, but it was mostly because I was swamped with preparations to my travel to Germany. I am writing this report in Germany right now.

As for the current state of the error reporting project. I believe it is ready to be released. I dont have any more clear ideas about how to make the system better. I have created alot of documentation and the server tests cover almost all of the code. I was actually able to do some refactoring relying on those tests safely.

I am creating another pull request with the latest changes. They are not much as stated earlier. I am going to ask the mailing list about what they think one last time. According to google next week is for docs since the soft deadline is tonight. I already created most of the docs but I think I have a little bit of undocumented code that may need some explanation.

I will be talking to my mentors on where to go from here and whether they think I should be doing something important at this time. But till then I will be getting a care free couple of days to enjoy Berlin. Hopefully I will also get the component merged into phpma/master soon.

Weekly update #13 [2/9-8/9]

This is technically my final update. I have finished with the testing and docs. I have also done some refactoring efforts of the phpma code as well as the server code. I still have some things to do but not much. I want the phpma branch to get merged soon.

I refactored the access control in the server. I also refactored the phpma code into a library and fixed failing tests. I added the documentation to all models and components. I left the controllers out for now since most of the functions are very simple. I may add some after the soft deadline. 

As for the error reporting server a few things are left like the server is not currently strong enough against a hostile attacker. I sanitize everything. and cakephp has a good security model but I believe I can make it stronger against hostile input.

I created 2 pull requests but they still haven’t been merged. I will try to have them merged this week. Next week I will be in Germany so I am not sure how much work will I be able to do there at first.

It has been a great summer working on this project. I am happy its almost complete 🙂

Weekly update #8 [29/7-4/8]

This week is the first week of a two week task to handle related reports and grouping of similar reports. It is also the week that we had our first deployment.

I deployed in order to get more feedback on how everything is going. Sadly the responses were not exactly overwhelming but I have a somewhat good view about how I have decided to proceed.

This week I worked on the js line translator and added url sanitization to most urls in the error report. This helps with two things. First is the increased privacy of the reporters, and second is that there is less difference between similar bugs just because they happened at different hosts.

I have also changed the current schema so that we have reports and incidents. I have moved some of the functionality to the incidents model. I have also changed the views and helpers to handle the new schema. However I faced some problems in how to display certain parts of the incidents. since the incidents do not have a view I need to display the relevant information in the reports view. The most problematic pieces of info is the steps leading to the error and the stacktrace.

They need to be displayed but caution must be taken since they may be 1000s of them. I am currently displaying all the steps at the bottom of the report. As for the stack traces I have not decided how I am going to handle them. I have an idea but I am putting it off incase another idea comes to me. The idea is to have php compare the stack trace to all the stacktraces of the related incidents in the database to find if it is the same as another stack trace of a previous incident, otherwise it is marked in the database as a new stacktrace and is displayed in the reports view.

Another problem is displaying all the different stacktraces. I am not exactly a great designer so I might not be able to create a view that is usable but doesn’t waste a lot of space in the page. We will see about it but I am not sure yet.

I have also yet to handle the manual marking of reports as the same. So for next week I am going to handle that, as well as add a view for the stack traces and maybe even create an html view for incidents and hopefully end up with something better than the last time I tried it.

I will not yet merge my current work into the master branch since it is not finished yet. I will wait till the task is finished to submit a pull request so that the deployment server gets the entire story at once.

Speaking of deployment, I was also able to get the checkstyle errors from around 900 to 43 or 57 depending on who you ask. This was quite impressive. I will hopefully continue on this trend.

Also I might take a couple of days off for Eid at the end of next week. I will try to finish next week’s work before that. I am not exactly behind on my schedule so I don’t think that would be a problem

Weekly update #7 [22/7-28/7]

The task this week was simple. It was to submit bugs to the sourceforge bugtracker. I had already spent a long time trying to get an authentication token in an earlier week. I was able to use this authentication token this week to submit error reports.

Sourceforge uses OAuth V1 which was much more complicated than V2. I was able to use a component created for OAuth V1 for cakephp. I wrote my own component specifically for sourceforge api, similar to the one I did for github. It allows most of the logic to be out from the controller so that it is much simpler.

Since I noticed there is a lack of components relating to apis for cakephp, I wanted to to release them on a separate repo to help others who may need to do what I did. The components I created were a bit specific, not in a way that they can only be used on phpmyadmin or my project, but in that I only implemented the parts of the api I needed. The api is pretty big and while most of the functions are not specific to phpmyadmin, they do not come close to covering a large portion of the api.

Anyway I created code to allow me to get authentication tickets in the future for any application and user credentials so that when the deployment time comes I can easily generate proper credentials other than the ones I was using for testing.

I also finished the line number translator in the phpmyadmin repo. I went with a much cheaper way than what I originally planned. It works great I just have to make it more resilient incase the line count cache file is missing.

Next week is one of the hardest tasks not because of too much coding is required but because I will be dealing with hypotheticals. I am expected to find some sort of measure to group reports together that is as accurate as possible with no way to test it in the wild. There are a million ways it can go wrong and I wouldn’t know until we actually deploy and test it.

I will be trying out the different ways a stacktrace can appear (it changes depending on how the code containing the exception was loaded). I will try to find some way to group them without alienating too many similar reports. We will see how it goes.

Weekly update #4 [1/7-7/7]

So this week is the first week with the server side component. I am using cakephp as the server software. I was able to get the requests sent by the client side component as well as store them in the database. I found many problems with the sending code of my client side component but after hours days of trying I was able to resolve them. I was also able to use datatables for the listing of the error reports.

Data tables allows for search, filter and sorting of data and I was able to interface it with the server side so that not all the data would be loaded into the page at once since the number of reports should increase rapidly. Currently you can search, sort as well as filter by status and phpma version as well as by error report id.

Currently there is no show page for the error report. That is left for another week. The tasks for next week should have been github integration. I actually started early and finished most of it this week. However it turns out that we may not be using github as planned after all.

The reason I chose github was for two things. I can use it to get user info such as the name and email address so as to forgo the registration page. Also I was thinking of using its issue tracker with the error reports.

However the phpmyadmin developers do not use github’s issue tracker, they use sourceforge. It was suggested that I use that issue tracker rather than github’s. I believe this to be the better choice but as it was not in the plans I wasted some time implementing a github authentication and api component for cakephp. And sadly the way sourceforge does oauth is different from the way github and all the other oauth providers do it so a part of the code cant be used.

But that is not the problem. The problem is that the authentication part of the api is not well documented. I tried it out but it seemed to give me a server 500 error. I was not able to understand why since no relevant error message was displayed to me.

I have submitted a ticket with sourceforge and hopefully they will reply asap so that I can start working on their authentication system. Another thing is that I am not sure whether to offer github integration with sourceforge or only the latter. Github gives me access to things like name and email which allows me to keep logs of the activities done on a report by the developers. It also gives me access credential to check whether the user is a core developer or not by checking if he has commit access on the main repo. This would allow me to do some access control without an admin system to grant users permissions.

but since we would be mainly using the sourceforge issue tracker and there will be nothing to do with a report except to submit it to sourceforge as a bug or mark it as related to another report so that they are grouped together. if there will be no commenting on reports or subscribing to them then emails and names are of no need and we do not need github authentication. however I have still to discuss this with the core phpmyadmin developers

We won !!

One of our main courses this semester is the Software Engineering course. Each tutorial group was a fictional company and divided further into 4 teams. Each tutorial group had a real client and a real project to create by the end of the semester. The aim of the course is to teach us collaborative development techniques as well as agile development.

A competition between the companies was created and the winner would be the one who best followed the process and delivered the best end product. All the projects were web development related and targeted the middle east, specifically Egypt.

The development was divided into 2 sprints of length 2 weeks. The project was divided into stories in a backlog. They were given points according to difficulty and were assigned to one of the 15 developers in the company. Every story had to be well documented, with unit tests and a sequence diagram describing it. The code had to follow code conventions and the story had to be peer reviewed and verified before merging it into the master branch.

Our project was a game with a purpose. The aim was to crowdsource arabic translations and then provide statistics to developers who are looking for the best translations targeting a certain country or age group. We also created a script to automate the translation of your page based on the country of the user of your website. The public repo can be found on github.

The competition was fierce but we persevered and as the title already revealed we won. The course was a resounding success and many would-be developers learnt the ropes of how to create a large software project successfully.

Our team at the award ceremony. I am in the back row, second from the left

Our team at the award ceremony. I am in the back row, second from the left