Dan Olsen’s Weblog

Well, after fighting DBXML once again I finally have it installed and working with Python. Once again, time to move forward and see if the issues I have been having have disappeared.

Technorati Tags: databases, dbxml, python

It is time to make some progress on this project. As I stated before, I was having some problems with mod_python and getting it to parse XML files. I have decided, by suggestion of Dr. Windley, to forget about mod_python for now and use cgi. I have now started implementing it and hope in the next day or two see if DBXML 2.0 fixed the previous database problems.

The XML database is up and running, I have tested it and have started looking into some of the new syntax. Most of the syntax changes that I have come across have to do with opening databases and basic manipulation. The new syntax is the kind of syntax I thought was strange that they did not include in version 1, things such as an open command for a database instead of a single command to open a new database or create a new database and other small things like that.

I have begun the process once again of building ROSE and have run into a minor problem where mod_python says that xml.dom.minidom does not have a parse method. That’s a little odd since I got the xml parser working the same way previously. I will continue to work on that today. I am not yet at the point to see if version 2.0 of the database will resolve the problems I encountered earlier.

Now that my classes are done I can focus my time back on the ROSE project.

I have downloaded and installed the new release of the Berkeley XML database. I was able to install the database without any hang-ups which was nice after not getting it to install properly with all the early access versions.

Once I got the database installed I tried installing the python wrappers. This took a couple hours a work trying to get all the shard libraries installed and working properly.

I now currently have everything currently installed and working. However, I am now having some problems with opening databases. I have tried following some of the examples that came with the database but it seems that when I create a container with a database that already exists and throws an error. I am currently checking that out and figuring out all the syntax changes they made.

Sleepy Cat has released the newest version of the Berkeley XML database (2.0). I will be looking to install the new version of the database on our servers in the EC Lab. I am hoping that the new version will remedy some of the problems I have been having with the ROSE project.

I have downloaded the pre-release of Berkeley XML Database 2.0. However, I have not been able to get it to install. For the 2.0 version of the database, Sleepy Cat has bundled all the applications that are needed for the install into the DBXML bundle. The bundling of all the packages makes it easier to install since you don’t have to install the individual packages yourself. Well, that’s if you can get it to install. All you have to do is run the script they provide you with and then everything should work. That is not the case here. I have been in touch with Sleepy Cat and they are looking at the output I sent them from their script to see if they can determine the reason why the database won’t install.

After not getting much help on the mod_python mailing list I went ahead and reinstalled Python and mod_python. The reinstallation seemed to do the trick and mod_python is up and running. I also had some slight problems with Berkeley XML database but it seems to be working now. Now I will continue to try to find a way to get the ROSE project to work with the XML database.

At the moment there hasn’t been a whole lot of progress. One problem at the moment is that I am having problems with mod_python and I am trying to get some answers on the mod_python mailing list. I am also researching other databases that we may be able to use to resolve the current database problems.

UPDATE:

We have found that the sem_wait error we have been getting when installing mod_python is caused by libpython2.3.a. To fix this problem we are going to reinstall Python.

I am continuing to have problems with the Sleepy Cat Berkeley XML Database crashing when trying to put certain feeds into the database. I have looked into MySQL and have not been able to find a good way to use MySQL with RSS feeds. I have been looking into other XML databases and have yet to find another database which supports Python.

If any one out there knows of a good, reliable, open source native XML database please let me know.

We are supposed to get word from Sleepy Cat sometime in the near future about Berkeley XML DB 2.0.

I have decided to post my design of the basic Rose system on our lab wiki. This way the design and how the system should work is easily viewed on the web for those who are interested.

The Rose project is nearing completion in it’s first stage. The project has been making slow progress as of lately due to problems between the program, which is written in Python, and the Berkeley XML database.

At the moment there is a piece that currently updates the RSS feeds in the database. This part of the project is not having any problems at the current moment. The part that is having the problems is the part that takes an OPML file and checks for the feeds in the file to see if they are in the database. If the feed is not in the database then it is retrieved over the internet and inserted into the database. It seems to work on a majority of feeds but crashes on a few feeds. I have been working with the folks at Sleepy Cat to see if we can find what the problem might be.

To be a little more specific as to what the problem is I will explain that below.

When the program reads in the RSS feeds I check them against the database, if the feed is not in the database then the program will retrieve them as stated above. However, on some feeds the program just quits with the message “Aborted”. I have narrowed the problem down to the putDocument call on the XML database. I have then taken the XML that is supposed to be put in the database and put it into a small program that implements the same procedure as in the main program. The feeds that crash the program work fine so it is not a problem with the structure of the XML. The problem lies somewhere within the rest of the program. I am looking into the possibility of the problem occurring due to the loop that goes through the array of feeds.

I am continuing to debug the process and find why certain feeds crash the program. I am also looking into other implementations that I may be able to use to get around the problem. I will post more of my implementation ideas I sort through the problem.

I will begin a new category dedicated to the ROSE project I am working on in Dr. Windley’s research lab. The category will have its own RSS feed which can be accessed at http://www.dan-olsen.net/weblog/rose.xml. I will post weekly reports on the progress of the project.