The following entry was posted on the RiP Dev blog today...
Previous Installer had another one major technical issue - it was using Property Lists (plist) files to store the database of available and installed packages. While this was very convenient for us (as programmers) as reading and writing to it required almost no effort, there was numerous complications caused by this design decision.
First, saving and loading property lists with the number of packages available across numerous repositories was long, simply because each property list file could get as large as 10-20 megabytes, and parsing that when Installer is starting up was not an instant operation. This problem was partially eliminated in Installer 3.1 and beyond when we have moved to binary format property list files from the plain XML - they take less space on disk and load and save faster.
Second, when the database with all available packages was loaded, it was kept in memory. The iPhone has strict rules about memory usage - so if you had too many sources or packages in the database, the Installer may have been closing due to not enough RAM conditions.
Third, searching through an array of, say, 1000 packages was slow - it had to run through all of them in memory one by one to find matches. Of course the ARM processor used in the device is fast, but either way that required time.
So Installer 4 uses sqlite database for all it's stuff - which means indexing and searching is fast, the index is not loaded all in RAM and only accessed on demand. The startup is significantly faster because of that.
Next thing I am going to tell you tomorrow is how the packages itself have changed.
And to answer a common question, we're getting there with the release - tomorrow the Installer may hit the private beta, depending on how well the work progresses.